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The EEF aims to raise the attainment of children facing disadvantage by: 


e identifying promising educational innovations that address the needs of disadvantaged 
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e evaluating these innovations to extend and secure the evidence on what works and can be 
made to work at scale; and 

e encouraging schools, government, charities, and others to apply evidence and adopt 
innovations found to be effective. 
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Executive Summary 


The project 


The SPOKES (Supporting Parents on Kids Education in Schools) programme is a ten-week intervention 
for parents designed to help struggling readers in Year 1. The programme teaches parents strategies 
to support their children’s reading such as listening to children read, pausing to let them work out words, 
and praising them when they concentrate and problem-solve. 


The programme was based primarily in Plymouth and was conducted with six cohorts of children and 
parents, one cohort each term from Spring term 2012 to Autumn term 2014. This evaluation was 
designed to assess the impact of the programme on children’s reading outcomes. It was a randomised 
controlled trial involving the parents of 808 children from 68 primary schools. Parents of Year 1 children 
identified as ‘struggling readers’ were recruited through their child’s school to participate in the project. 
Parents of the intervention children participated in ten weekly SPOKES sessions over one term and 
parents of children in a comparison group received books and newsletters. The impact evaluation 
measured the impact of SPOKES—on children’s literacy (letter identification, word identification, and 
phonetic awareness) and on a range of social and emotional outcomes—at the end of the programme 
and at six- and twelve-month follow up points. The process evaluation was designed to collect parents’ 
views and experiences of SPOKES, to help to explain the findings from of the impact evaluation, and to 
provide feedback to inform the future design and delivery of the programme. 


The programme was developed as part of a Department of Health grant agreement by Professors 
Stephen Scott and Kathy Sylva. This project was funded by the Education Endowment Foundation. 


CW al Oe) ated [VEN (o}at-y 


1. There is no evidence that SPOKES had an effect on the standardised reading outcomes or the 
social and emotional outcomes of the participating children, overall. 


2. However, the follow up results for attainment suggest that SPOKES had a positive impact on 
some aspects of reading for boys over the longer term, equivalent to between three and seven 
months of additional progress. 


3. Parents who participated in the SPOKES programme were very positive about it and believed 
that their children’s literacy was enhanced from their participation. 


4. Factors that may have contributed to the lack of overall impact include a low attendance rate 
and a sample that included children of higher ability than the programme was designed for. 


5. In addition to the pre-agreed analysis, the evaluation team also assessed the impact of 
SPOKES on Key Stage 1 scores for the subgroup of pupils who had reached KS1. This 
analysis is promising and the positive results warrant future research and further analysis of all 
the participants at the end KS1. 


How secure are the findings? Se euily EaUnG awarded as 
part of the EEF peer review 


These findings have moderate security. The study was a large, well-designed, two-armed randomised 
controlled trial with randomisation at the pupil level. The trial was classified as an efficacy trial, meaning 
that it sought to test whether the intervention can work under ideal or developer-led conditions. For the 
pupils included in the analysis, those whose parents were allocated to receive the intervention were 
observed to be similar to the pupils in the comparison group. However, 22 % of the pupils were not 
included in the final analysis because they did not complete all the tests at the beginning and end of the 
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trial. This is considered to be a relatively high level of attrition and reduces our ability to be sure that the 
testing is an accurate reflection of all the pupils in the sample. 


What are the findings? 


In this efficacy trial there was no evidence, for the intervention group as a whole, that SPOKES had an 
impact on the outcomes specified in the trial protocol at post-test or at six- and twelve-month follow-up. 
A positive effect for SPOKES was found on KS1 teacher-based literacy measures; these, however, were 
not pre-specified as outcome measures. 


Analysis of the results for boys only shows that, compared to other boys, those whose parents took part 
in SPOKES experienced about three months of additional progress on word identification and phonetic 
awareness at the six-month follow-up, and about seven months of additional progress on phonetic 
awareness at the twelve-month follow-up. In each case, the positive result would be unlikely to have 
occurred by chance. The positive results are not conclusive because follow-up results were not available 
for all boys in the trial, and because positive results were not found for all of the three outcome measures. 
However, given that SPOKES is intended to have an impact on pupil attainment by changing parental 
behaviour, some positive results at both follow-up points suggests that SPOKES may have improved 
aspects of reading for boys over the longer term. For children eligible for free school meals there was 
no evidence of an impact on literacy or social and emotional outcomes. 


The average attendance rate indicated that only 57% of the parents participated in five or more of the 
ten sessions, which was judged by the development team to be the level of attendance required to be 
considered a ‘completer’ of the SPOKES programme. This low attendance rate might have contributed 
to the fact that there was no evidence of impact on the overall group. 


The process evaluation indicated that most of the intervention parents who were interviewed reported 
liking and benefitting from their participation in the SPOKES programme. At both immediate post-test 
and subsequent follow-ups they believed that it improved their children’s reading. The table below gives 
the average effect size and estimated months’ progress at post-test across the three primary outcome 
measures compared to the control group. 


How much does it cost? 


The cost for the programme implementation in this study was approximately £804 per place, which could 
be considered per child, per parent, or per family. These costs are predominantly staff costs to 
implement the weekly SPOKES sessions. 


Sues Estimated 
Outcome oe months’ pecuuey Cost rating 
Coxe) avi Co (=Valex-) relat ale} 
: progress 
aly are-1)) 
0.08 
Letter ID 679 (-0.07, 0.23) +1 88 LELE 
0.05 
Word ID 659 (-0.10, 0.20) +1 88 LEEE 
Word Attack 660 aoe +11 aaa ERE 
(-0.13, 0.18) 


1 Since this report was published, the conversion from effect size into months of additional progress has been 
slightly revised. If this result was reported using the new conversion, it would be reported as 0 months of additional 
progress rather than +1. See here for more details. 
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Introduction 


Intervention 


SPOKES—Supporting Parents On Kids’ Education in Schools—is a series of ten weekly sessions to 
teach parents strategies for supporting their children’s reading. It is designed for the parents of struggling 
readers in Year 1. Examples of strategies parents are taught include listening to children read, pausing 
to let them work out words, and praising them when they concentrate and problem-solve. It also 
encourages discussion of story content and plot. 


The SPOKES programme evaluated in the current study is based on an earlier version of the SPOKES 
programme which was a combination of support for parents in improving their children’s social behaviour 
(drawn from the Incredible Years programme) and helping parents to support their children’s learning to 
read. The SPOKES programme implemented in the current evaluation was different to that previously 
evaluated: it was shorter than the original SPOKES programme and focused primarily on teaching 
parents to support their children’s oral language and reading. The SPOKES programme evaluated in 
this EEF study aimed to provide parents with the specific skills they need to support their young 
children’s learning to read words and comprehend story meaning. 


The parenting sessions were implemented by the Plymouth Parent Partnership—a social service 
agency located in Plymouth—and delivered by pairs of group leaders consisting of an educational 
psychologist and a Parenting Programme Facilitator. The group leaders were trained to implement the 
programme by the programme developers and attended weekly supervision sessions on filmed groups 
(see ‘Implementation’ section in the process evaluation). 


Background evidence 


There is strong evidence that parenting skill is one of the strongest predictors of positive outcomes for 
children, both behavioural and cognitive (Gardner, 2013). However, the evidence about how to improve 
parental support for children’s academic success, and what kind of involvement improves children’s 
attainment, is much less clear. 


The early version of SPOKES, cited above, was evaluated in a randomised study of 108 five- and six- 
year-old children selected from eight primary schools in inner-city London. The selected children were 
identified as being at risk of social exclusion through antisocial behaviour. The study focused on 
improving children’s behaviour and literacy and was found to be effective in improving the word reading 
(decoding), writing skills, and behaviour of these emotionally dysregulated children (Scott, et a/., 2012; 
Sylva et al., 2008). The intervention was delivered between 1999 and 2001 before widespread phonics 
teaching was rolled out. The early version was longer than the version of SPOKES evaluated here and, 
through the inclusion of dimensions of the Incredible Years parenting programme, focused more on 
improving the children’s social behaviour and literacy rather than focusing primarily on literacy 
attainment like the updated SPOKES programme implemented for this evaluation. 


The programme evaluated in this current study is referred to as SPOKES throughout the report. It 
focuses on improving children’s literacy skills, although children’s social behaviour and the parenting 
behaviours of parents or carers were also measured as secondary outcomes. The literacy strategies of 
SPOKES are based on the Pause Prompt Praise (PPP) approach initially developed in New Zealand in 
the 1970s to train parents as tutors (McNaughton, Glynn and Robinson, 1987) and on broader ‘whole 
language’ approaches developed by Clay (1991). 


The current version of the SPOKES programme has not been widely implemented. However, the 
previous study, and wider research indicating the strong influence of parenting skill on children’s 
academic achievement (Gardner, 2013), suggested that an efficacy trial of the SPOKES programme— 
a trial of the programme under ideal and controlled conditions as opposed to a real-world 
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implementation—was warranted. If proved effective, it would contribute to policy change to help 
struggling readers. 


Evaluation objectives 


The impact evaluation was designed to determine whether the participation of parents in the SPOKES 
programme had an impact on: 


1. children’s early literacy skills; 
2. children’s social behaviours; and 
3. parents’ parenting behaviours. 


A ‘light touch’ process evaluation was conducted alongside the impact study. This was primarily to 
determine if there were any discernible factors that might explain the outcome findings of the study, 
such as the level of participation of the parents and their implementation of the programme. The process 
evaluation was also designed to provide feedback to the development and implementation teams to 
help guide refinements of the programme and its delivery in the future. In addition, a random sample of 
one third of the parents in the intervention group was interviewed by researchers by phone with the aim 
of understanding parental perceptions of SPOKES. 


Project team 


Louise Tracey, PhD. Lead Evaluator. Dr Louise Tracey is a Research Fellow in the Institute for 
Effective Education at the University of York. She conducts evaluations of literacy and mathematics 
programmes in primary schools using experimental and quasi-experimental methodologies. Dr Tracey 
recently conducted an evaluation of the Success for All programme, a longitudinal matched study 
involving 40 primary schools over four years and literacy assessments in Reception classes as part of 
the Born in Bradford cohort study. 


Bette Chambers, PhD. Co-investigator. Professor Bette Chambers is Director of the Institute for 
Effective Education at the University of York. Professor Chambers develops and evaluates effective 
practices in early childhood and literacy education and promotes the use of evidence-based practices. 
She has authored or co-authored numerous articles, books, and practical guides for teachers. 


Tracey Bywater, PhD. Co-Investigator. Dr Tracey Bywater is a Professor in the Institute for Effective 
Education at the University of York, focusing on enhancing parental input in supporting children’s 
success and long-term positive outcomes. She is also an Honorary Senior Research Fellow in the 
School of Psychology, Bangor University, Wales. Dr Bywater has led on many randomised controlled 
trials (including Incredible Years) to improve behavioural, cognitive, and health outcomes in children and 
their parents or carers. 


Louise Elliott. Louise Elliott is the Data Manager at the Institute for Effective Education where she 
manages all database organisation, data entry, cleaning, and descriptive statistical analyses conducted 
in the research work. 


Ethical review 


The evaluation team obtained ethical approval from the Department of Education, University of York 
Ethics Committee in January 2013. Head teachers signed an agreement outlining the main 
commitments of the three parties in the study: the school, the project developers, and the evaluators. 
The evaluation team provided information and consent forms for parents/guardians (see Appendix A). 


The assessments were conducted in primary schools by fully trained, Disclosure and Barring Service- 
checked assessors hired, trained, and supervised by the lead evaluator. 
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Data were managed in accordance with the Data Protection Act (1998). The trial database is securely 
held and maintained on the University of York’s research data protection server, with non-identifiable 
data. Confidentiality is maintained and no one outside the trial team has access to the database. Data 
were checked for missing values and/or double entries. All outputs were anonymised so that no schools 
or students could be identified in any report or dissemination of results. 


Trial Registration 


This trial was registered at http://www. isrctn.com/ISRCTN13234022. 
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Methodology 


Trial Design 


The trial was a two-armed, within-school, pupil-randomised efficacy trial. It was designed to assess the 
impact of the programme on reading outcomes of struggling readers in Year 1. The programme was to 
be run for six cohorts of parents, one cohort each term, starting in the Spring term 2013. Parents of 
struggling readers were to be identified and recruited through their child’s school. The Plymouth Parent 
Partnership was responsible for recruiting the schools and the schools were expected to take part in 
recruiting parents for two cohorts of the study. Five cohorts were recruited solely from the Plymouth area 
and one cohort extended the geographical area to include the surrounding region. This was to ensure 
that recruitment was not taking place from the same pool of children between cohorts—that struggling 
readers from Year 1 in each school were only identified once per academic year. 


In each of the six cohorts, it was anticipated that approximately eight pupils in Year 1 per school would 
be designated as struggling readers. Where siblings were identified it was intended that only one would 
be selected for the outcome measures. Forty-five primary schools were anticipated to take part—each 
school participating in two cohorts, and identifying eight eligible pupils for each—making a total of 720 
pupils. Pupils were to be randomly assigned within schools to the SPOKES programme or to the control 
group (approximately 360 in each). The parents of those assigned to the intervention condition were 
offered the ten-week SPOKES programme and those in the control condition received children’s books 
and newsletters. The control group was anticipated to provide an ideal counterfactual for the intervention 
as being within the same school, control group children would receive the same education beyond the 
SPOKES programme; furthermore, children within schools tend to share more characteristics than 
children from different schools. The Institute for Effective Education (IEE) conducted the random 
assignment. 


The children (control and intervention) had their vocabulary assessed in school prior to the programme 
commencing, and their literacy assessed (letter identification, word reading, and phonetic awareness) 
at the end of the programme, at six months, and at twelve months for the cohorts that had reached those 
milestones. Additional child and parent behavioural measures and parents’ perceptions were also 
collected at those time points. 


Eligibility 


The ‘struggling reader’ definition was intended to include children who had been identified as having 
difficulties in their literacy learning using the previous year’s teacher-completed Early Years Foundation 
Profile 5+ (EYFSP). This measure was chosen as the EYFSP is a statutory requirement at the end of 
the Reception year and therefore would have been documented for all children. However, as schools 
often felt that the EYFSP was not a good reflection of the child’s current ability due to being completed 
at the end of Reception year, this was changed so that a ‘struggling reader’ was defined as a child 
identified by the teacher as not being where they ‘would want or expect the child to be in literacy’ based 
on their knowledge of the child. Therefore, both the EYFSP and the daily experiences of the Year 1 
teacher were used to assess eligibility for the programme. This means that, as the schools were 
responsible for identifying participants, the identification process and criteria may have been subject to 
differences between schools. 


In addition, in order to be eligible for the study, a child’s parents had to agree to (and be able to) 
participate in the SPOKES programme if their child was subsequently selected to be in the intervention 
group. While this may have resulted in some selection bias this would have held true for both the 
intervention and control conditions. 
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Intervention 


The SPOKES programme aims to give parents the skills they need to help their children learn to read, 
particularly children who are struggling readers. It consists of a series of ten group sessions led by two 
trained group leaders (an educational psychologist and a Parenting Programme Facilitator) following a 
programme manual to ensure uniformity of delivery. 


During the programme, parents learn strategies to support their children’s literacy learning, these include 
Pause Prompt Praise and ‘whole language’ strategies. In the PPP approach parents are shown how to 
respond when their child does not recognise a word. Instead of intervening immediately, which can 
encourage dependence, parents are taught to wait for five seconds to allow the child time to grapple 
with the problem. If the child did not produce the right answer, parents are encouraged first to offer a 
prompt before revealing the correct answer. Parents are also taught how to give effective feedback by 
praising a specific reading behaviour. In the whole-language approach parents are encouraged to 
discuss books with their child, to relate the story to their everyday experience, and to support their child 
to anticipate what might happen next. 


Supporting these core components, most sessions consist of practical activities (for example, making 
puppets for re-telling stories, word games, making up novel story endings), group discussion, role play, 
homework, and demonstrations of reading with children. The programme promotes collaborative 
learning to help motivate parents to learn new strategies. In addition, a library visit and a one-to-one visit 
(usually in the home and with both parent and child present) are included in the SPOKES schedule 
(Sylva et a/, 2013). Books and games are made available for parents to borrow and share with their 
family between sessions. Parents are also encouraged to make use of print available in their 
environment to further support their child’s reading skills. 


For this study SPOKES was delivered by the Plymouth Parent Partnership (‘the Partnership’), an 
organisation that helps support parents, especially those whose children have learning difficulties. The 
Parenting Programme Facilitators were provided by the Partnership, and the educational psychologists 
were provided by Plymouth Community Psychology Service. All SPOKES group leaders were provided 
with a three-day training programme that focused on techniques for training parents in the skills they 
need to support reading at home. An additional three-day training course in the Incredible Years 
parenting programme was also provided prior to this to all group leaders who were not already trained 
in the Incredible Years. They also attended weekly supervision sessions conducted by the developers 
and a Partnership disseminator. These sessions were designed to help group leaders enhance their 
knowledge and skills to deliver the SPOKES programme and maintain fidelity. 


Parents of the control group children received children’s books to share with their children and 
newsletters containing information about activities to do with their children, based around the relevant 
free book given out at the time. The Partnership contact telephone number for the SPOKES programme 
was also featured in each of the newsletters. These resources were sent home in children’s book bags 
over the ten-week period and were an added incentive to both schools and parents to participate in the 
study. 


Outcomes 


The primary tests for the impact analyses were administered by assessors, hired and trained by 
evaluators from the Institute for Effective Education. They all had Disclosure and Barring Service (DBS) 
checks, received training and coaching from the IEE, and were blind to the condition that each child was 
in. Blinding to condition was easy to maintain as it was the parents who participated in the treatment, 
not the children. Results of these tests were supplemented by the results of national routinely collected 
assessments recorded on the National Pupil Database (NPD). Additional measures were completed by 
the teacher or the parent (facilitated by an IEE assessor). 
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Measures were as follows: 
1) Completed by the child: 


a) Woodcock Reading Mastery Tests (revised, WRMT-R, Woodcock, 1998). The co-primary outcome 
measures were the Letter Identification, Word Identification, and Word Attack sub-tests of the WRMT- 
R. These were administered at post-intervention; for cohorts one to five at six months post-baseline, 
and for cohorts one and two at 12 months post-baseline. They are standardised, individually 
administered tests of reading designed to assess letter recognition, word recognition, and phonemic 
awareness. These capture reading skills that beginning readers are expected to learn. Reliability 
coefficients for these sub-tests range from 0.94 and 0.98 for the target age range. 


b) British Picture Vocabulary Scale (second edition, BPVS II, Dunn et al., 1997). The BPVS was 
administered at baseline only and formed the pre-test for each participating child. The BPVS II is an 
individually-administered, norm-referenced test of receptive vocabulary for Standard English. It is a 
standardised test showing the extent of English vocabulary acquisition. The BPVS was chosen for the 
pre-test because children’s oral language correlates highly with children’s 1Q and subsequent reading 
ability. Cronbach’s Alpha, which is a measure of internal consistency, is high: for the Year 1 age group 
itis 0.96. 


c) Year 1 phonics screening check. This is a compulsory check administered in schools in the summer 
term to assess Year 1 pupils’ phonics decoding skills. This data was collected from the NPD for all 
participating pupils (cohorts one to five). Data was not available for cohort six at the time the analysis 
was done. This is a measure of importance to parents and schools as it is seen as a marker of phonic 
competence and was chosen to assess children’s phonemic awareness at a specific time point in their 
school career. 


2) Completed by the teacher about the child: 


The Strengths and Difficulties Questionnaire (SDQ, Goodman, 1997), with impact supplement. 
Additional detail on the children’s behaviour and emotional wellbeing was collected using the SDQ. This 
was completed by teachers at post-test and subsequent follow-up phases. It is a 25-item behavioural 
screening questionnaire covering five dimensions of children’s behaviour (age 4—16): conduct problems, 
inattention-hyperactivity, emotional symptoms, peer problems, and pro-social behaviour. The impact 
supplement asks teachers about any difficulties they feel the child may have. The different dimensions 
were to be looked at separately to assist in identifying whether the programme had any impacts on 
children’s social behaviour, and for which children and behaviours the SPOKES programme was most 
effective. 


3) Completed with the parent: 


The PARYC (McEachern et al., 2012) parenting questionnaire. This is a relatively new measure with 
three subscales: supporting positive behaviour, proactive positive parenting, and setting limits. This 
measure was included to assess the parenting skills that the SPOKES programme is designed to affect. 
The first two subscales were used as they more accurately reflected the underlying principles of the 
SPOKES programme and were piloted in autumn 2013. The PARYC items are rated via a seven-point 
Likert scale. The PARYC was normed specifically with the parents of children aged five and six. 
Cronbach’s alpha for Supporting Positive Behaviour is 0.78 and for Proactive Parenting is 0.85. 


In the study, this assessment was to be administered in person at baseline by the Partnership home 
visiting team during pre-programme home visits, with a post-test assessment administered with all 
parents (one main carer per child) by telephone interview at post-test. The Partnership unfortunately did 
not have the staff resources to conduct these initial home visits and so the PARYC was collected from 
parents during a 15-minute telephone interview by the IEE evaluation team at post-test and subsequent 
follow-up phases. 
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In addition, although not specified in the protocol, the Key Stage 1 assessment literacy results were 
available for cohorts one and two at the end of the study. These are national formal assessments, 
administered and scored by the children’s teachers and taken by all children at the end of Year 2. It was 
decided to include these measures in this report even though they were not specified in the protocol, as 
they are regarded nationally as a key indicator of children’s literacy development. The results of the 
reading and writing elements of the assessments for these children were obtained from the NPD. 


Sample size 


Using the specifications of the sample size outlined in the Trial Design section above (approximately 16 
pupils per school from 45 schools across six cohorts), the IEE estimated the statistical power of the 
planned analyses using Optimal Design software. Assumptions were as follows: 


e Pre-post correlation (Squared): 0.49 
e _Intra-class correlation: 0.15 
e Criterion for statistical significance: p < 0.05 


Based on these assumptions it was determined that a minimum detectable effect size (MDES) of 0.20 
could be detected with a probability of 0.80. 


Due to the small numbers of parents recruited in some schools and concerns relating to retention, the 
Plymouth Parent Partnership over-recruited to the study, and 68 schools in total took part over the 
lifetime of the project. This resulted in 808 participants—a number that the team felt would allow for any 
attrition. As outcomes were linked to the individual child rather than the school (children could be 
retained in the programme if they moved schools within the Plymouth area) attrition was anticipated to 
be low. 


Randomisation 


Using a random number generator in Excel, the IEE data manager conducted the randomisation of the 
subjects who had been recruited by Plymouth Parent Partnership. For each cohort of participants, pupils 
were randomised to intervention or control conditions once parental consent had been received. Where 
possible, pupils were randomised within school. Because within-school randomisation was not possible 
where there was only one child per school, in schools with odd numbers of eligible children, the highest 
possible even number of children was randomly selected and randomised within school and the 
randomly selected ‘extra’ child was added to a pool and randomised a second time. Where siblings were 
identified, only one was randomised to be included in the outcome measures. The Oxford development 
team intended to conduct more in-depth research with one third of the sample of the children and parents 
involved in the project. The IEE team randomly assigned an additional third of the parents in the 
intervention group to be interviewed. Table 1 shows the allocation to condition of participants per cohort 
and the proportions assigned to participate in the York and Oxford administered measures. Only the 
York measures are reported in this report. 
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Table 1: Allocation of participants for data collection by York or Oxford 


York testing: intervention (4 participants) York testing: control (2 participants) 
York 
interviewing Oxford Oxford 
1/3 of testing 1/3 of | testing 1/3 of 
intervention intervention control 
parents 
Analysis 


The evaluation was intended to be a within-school design (among 45 schools) comparing outcomes 
between intervention and control, with 16 children per school (8 in each of the two cohorts each school 
was expected to participate in) with poor reading ability allocated randomly to the two conditions. The 
primary analysis was an ‘intention to treat’ design, meaning that even if parents dropped out or ceased 
attending sessions for any reason, associated outcomes would be retained in the main analyses. 
Analysis of covariance (ANCOVA) was specified as the main statistical analysis in the protocol, with 
baseline BPVS scores as the covariate. The model took into account the pre-test (BPVS), cohort, age, 
gender, and Free School Meals status, with the effect sizes calculated using Hedge’s g. The effect size 
was calculated using the Centre for Evaluation and Monitoring, University of Durham effect size 
calculator? using the observed standard deviation for each group. The means were estimated after fitting 
the model. School was not included in the model because of the lower than anticipated number of pupils 
per school recruited and retained in the study, and the lack of a relationship between the group a parent 
attended and the school their child attended. 


Because children participated in the study at different times of the year depending on the cohort they 
were in, standardised scores that take into account children’s age were used in the analyses of the 
primary outcomes (WRMT-R scores). Consequently, age was not included in the model for these 
measures but was for the social and behavioural secondary outcomes and the Year 1 Phonics 
assessment and Key Stage 1 reading and reading/writing results. 


Further analysis was conducted to explore whether the programme was more or less effective for FSM 
pupils compared to their non-FSM peers, and for girls or boys. Data on attendance at the SPOKES 
session for parents was collected by the Partnership, and FSM data was obtained from the NPD to 
inform this analysis. 


Additional analyses were also performed to include parental participation levels (dosage). The 
development team specified that attendance at five or more sessions was regarded as sufficient to be 
regarded as a completing the programme. In accordance with this, the intervention group was split into 
those who attended four or fewer sessions and those who attended five or more. The three-level 
treatment factor used contrasts to compare the controls with those attending four or fewer sessions, and 
comparing those attending four or fewer with those who attended five or more. 


Both the treatment by pre-test (BPVS) and the treatment by cohort interactions were included in the 
model. There was no evidence of a treatment by pre-test interaction being significant. The treatment by 
cohort interaction had occasional significant effects but with no consistent findings for the time point of 
outcome measure. So for ease of interpretation we decided to exclude both interaction terms. 


7 www.cem.org/effect-size-calculator 
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Process evaluation methodology 


A ‘light touch’ process evaluation was conducted alongside the impact study. This was primarily to 
determine if there were any discernible factors that might explain the outcome findings of the study, 
such as the level of participation of the parents and their implementation of the programme. The process 
evaluation was also designed to provide feedback to the development and implementation teams to 
help guide refinements of the programme and its delivery in the future. 


Data for this process evaluation was collected by the Partnership and included attendance data, parent- 
completed weekly evaluation forms collected at the end of each session, and an overall evaluation form 
for parents to complete at the end of the intervention. 


In addition, a random sample of one third of the parents in the intervention group was interviewed by 
researchers by phone. The goals of the interviews were to gain the parents’ perceptions of: 


e how many sessions they missed and what made their attendance difficult; 

e what would make it easier or more attractive for them to attend; 

e what specific features of the SPOKES programme the parents liked the most; 

e what they liked least and their suggestions for changes; 

e what SPOKES strategies or activities they implemented with their child and those they did not; 
and 

e what was their overall view of SPOKES. 


The interview questions are presented in Appendix B. For those participating in one of the first five 
cohorts, the interviews were repeated at six months, and for those in the first two cohorts, at 12 months 
to assess any longitudinal benefits. 


Process Evaluation Analysis 


Evaluation Forms 


The responses to the Likert-scale weekly and end-of-programme evaluation forms were input into an 
Access database and then transferred to SPSS for analysis. A researcher read through the open 
responses to the questions on the end-of-programme evaluation forms which were completed by the 
parents. She identified common themes, and created a coding frame. Key quotes for each code were 
noted. 


Telephone Interviews 


Responses to the telephone interview questions were input directly onto the computer using a specially 
designed Access database. This was then exported to Excel in preparation for a thematic analysis. A 
researcher read through the responses to the questions, identified common themes, and created a 
coding frame. Answers for each question were grouped and noted and a short summary of the key 
themes was created. 


Education Endowment Foundation 14 


SPOKES 


Impact evaluation 


Timeline 


SPOKES is a ten-week programme, which means that it lasts for approximately a full school term. The 
intervention was to occur over three academic years. The programme was run each term, with one 
cohort per term, for six terms starting in January 2013. Parents in the treatment group received the 
intervention for one term during the academic year when their child was in Year 1. There were six cohorts 
(see Table 2). 


Table 2: Intervention timeline 


School Term (and academic year) of intervention 


Cohort 1 Spring Term (2012/2013) 
Cohort 2 Summer Term (2012/2013) 
Cohort 3 Autumn Term (2013/2014) 
Cohort 4 Spring Term (2013/2014) 
Cohort 5 Summer Term (2013/2014) 
Cohort 6 Autumn Term (2014/2015) 


Recruitment took place in the preceding term for each cohort. This meant that there could be a long time 
lapse between recruitment to the trial and the start of the programme. Pre- and post-testing was 
undertaken as close to the beginning and end of each run of the programme as possible, although the 
timing did vary depending on the school term dates for the two academic years (for example, some post- 
test assessments for the Summer programme took place at the beginning of the following Autumn 
Term). Post-test assessments were intended to be scheduled within two weeks of completion of the 
programme. Where this was not possible the mean delay at follow-up was approximately four to five 
weeks. Six-month and twelve-month follow-ups were scheduled depending on when the post-test was 
administered. Again, some leeway was necessary to take into account the school term dates although 
this was the same for both intervention and control groups. This meant that there was a rolling 
programme of assessments in the schools over the time period January 2013 to March 2015 (see Table 
3). 
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SPOKES 


Pre-test* 
CA C2 3 C4 G5 C6 
Post-test* 
C1 C2 G3 C4 €5 C6 
6-month 
~upt 
follow-up C1 C2 C3 C4 C5 
— | 
12-month 
“ant 
follow-up C1 C2 


*Pre-test: BPVS 


* Post-test, six-month and twelve-month follow-up: WRMT sub-tests, SDQ, PARYC 


C = Cohort 
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Participants 


Recruitment was conducted through schools. As indicated above, school personnel, primarily Year 1 
teachers or the Year 1 literacy lead, were asked to identify struggling readers as defined in the ‘eligibility’ 
section above. Those children’s parents were then invited to coffee mornings or afternoon tea sessions 
at the school where more information was made available about the study and they were invited to sign 
up for the research. Any parents unable to attend were offered home visits where they could be provided 
with the information. Parents were required to agree to participate in the programme if they were 
selected, and to take part in the evaluation regardless of randomisation outcome. 


To ensure that pupils were not recruited from the same pool across rounds, schools could only take part 
in one cohort during the academic year 2012/2013 (cohorts one and two), and one cohort in the 
academic year 2013/2014 (cohorts three, four and five). All schools were eligible for cohort six as this 
was the only term in the academic year 2014/2015 that the programme was run and therefore the 
parents of Year 1 pupils would not be approached for recruitment to the programme at any other time 
in that school year. Plymouth Parent Partnership was responsible for recruitment. Following completion 
of parental consent, child and parent information were passed to the evaluation team’s data manager 
for random assignment to intervention or control groups. 


It was originally estimated that the parents of 16 pupils would be recruited per school across 45 schools, 
providing a total of 720 parent/pupil participants. Because there were fewer children per school than 
anticipated, more schools were recruited to participate (68 rather than 45) and more children (808 rather 
than 720). There were between 1 and 21 children recruited per school within each cohort. As indicated 
in Figure 1, in the final (analysed) sample (n = 627, 66 schools) there were between 1 and 33 pupils per 
school across the six cohorts. 
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Figure 1: Number of pupils per school in the final (analysed, post-test) sample 


Number of Schools 


123 4 5 6 7 8B 9 1011 121314 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 
Number of Pupils 


Tables 4-7 provide the characteristics of the participating schools. As can be seen, 4% (3 schools) were 
in areas with the highest band of deprivation as determined by the Income Deprivation Affecting Children 
Index (IDACI), and the same number were rated ‘inadequate’ by Ofsted. Sixty per cent of schools were 
in the highest two quintiles for proportion of children in receipt of Free School Meals (having18% of 
pupils or more in receipt of FSM). The Key Stage 2 reading achievement outcomes were fairly evenly 
spread across the reading attainment quintiles. Four schools were infant schools with the remaining 
being primary schools, and the majority were in urban locations, with only two being classified as rural. 


Table 4: IDACI rankings of recruitment schools 


ley-Tare| 


Highest 


Number 
of schools | 32 (47%) 8 (12%) 8 (12%) 6 (9%) 9 (13%) 2 (3%) 3 (4%) 
(%) 


Table 5: Ofsted ratings of recruitment schools 
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Requires 


OTT claret are] Inadequate 


Improvement 


Number of 


schools (%) 10 (15%) 46 (68%) 9 (13%) 3 (4%) 


Table 6: FSM quintiles of recruitment schools 


Lowest Highest 
quintile quintile 


Number of 


schools (%) 6 (9%) 11 (16%) 10 (15%) 21 (31%) 20 (29%) 


Table 7: Key Stage 2 reading attainment quintiles of recruitment schools 


Lowest Highest 
quintile quintile 


Number of é R 5 : ; 
schools (%)* 15 (24%) 10 (16%) 15 (24%) 10 (16%) 12 (19%) 


* 6 of the 68 schools did not have data available. 


Of the 808 participating families, 734 were included in the analyses. This represents less than 10% 
attrition between pre- and post-test. Figure 2 provides the overall participant flow chart (consort 
diagrams for each individual cohort are provided in Appendix C). The numbers in the participant flow 
chart are based on the number of children who completed the WRMT-R Letter Identification subtests at 
each follow-up. On occasion, a child might have been absent for one of the test administrations but been 
present for other data collection, hence the numbers do not add up to the total. If children were not 
available for the administration of the baseline assessment they were excluded from further data 
collection. Cases were excluded if children did not complete any post-test or follow-up assessment. 
Children who moved school were followed within area. 
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Figure 2: Consort Diagram—whole sample 


Assessed for eligibility (nN=821) 


Randomised (n=808) 


Excluded (n=13) 


Did not meet inclusion 
criteria (n=2) 


Declined to participate 
(n=11) 


Y 


Allocated to intervention (n=402) Allocated to control (n=406) 


Received allocated intervention (n=402) Received’ allocated intervention 
(n=406) 
Tested (n=369) - Tested (n=378) 
Not tested (n=33) Not tested (n=28) 
Excluded from analysis (n=41) Excluded from analysis (n=19) 


Analysed (n=328) Analysed (n=359) 


Tested (n=332) Tested (n=366) 

Not tested (n=70) Not tested (n=40) 

Excluded from analysis (n=38) Excluded from analysis (n=33) 
Analysed (n=294) Analysed (n=333) 


Tested (n=247) Tested (n=278) 

Not tested (n=71) Not tested (n=44) 

Excluded from analysis (n=33) Excluded from analysis (n=11) 
Analysed (n=214) Analysed (n=267) 


Tested (n=81) Tested (n=94) 

Not tested (n=42) Not tested (n=29) 

Excluded from analysis (n=13) Excluded from analysis (n=8) 
Analysed (n=68) Analysed (n=86) 
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Pupil characteristics 


Table 8 summarises the pupil characteristics of the sample, comparing the recruited sample pre- and 
post-test profiles in terms of gender, Free School Meals status, and pre-test attainment. There was little 
difference between the standardised pre-test scores in the two groups. In addition, there was little 
difference between the standardised pre-test scores for those who also gave post-test scores. The 
control group had a higher proportion of boys than girls, whereas the intervention group was fairly evenly 
split. The proportion of pupils ever eligible for FSM (having ever claimed FSM) was similar in the two 
groups. The profile for pupils completing post-tests was very similar to the pre-test, with the exception 
of those in the post-test group where those with lower baseline attainment were lost from the control 
condition due to withdrawal from the study. In contrast, for the intervention group pupils with higher 
baseline attainment were lost to follow up. 


Table 8: Pupil characteristics of the sample 


All pupils cohort 1 to 6 All pupils with post-test scores* 
(as randomised) (as analysed) 
Intervention level al ixe)| Difference Intervention lefeyaliKe) | Difference 
Pre-test 
standardised 100.92** 99.97** 0.95 100.60* 100.28*# 0.32 
score 
Gender 
Male 50.0% 60.1% -10.1% 50.3% 62.8% -12.5% 
Female 49.5% 39.7% 9.8% 49.7% 37.2% 12.5% 
Unknown 0.5% 0.2% -0.3% - - - 


FSM (Ever FSM) 


No 65.4% 68.5% -3.1% 71.8% 71.5% 0.3% 

Yes 24.9% 26.6% AZ 27.2% 27.3% 0.1% 
Unknown 9.7% 4.9% 4.8% 1.0% 1.2% 0.2% 
Saat 402 402 : 294 333 ; 


* All pupils with a valid pre-test and a valid post-test (Letter Identification) standardised score. 
** All pupils with valid pre-test standardised score. 


# ES = 0.031 (for the intervention group at baseline compared to the intervention group included in the analysis, i.e. those for 
whom a post-test score was established) 


## ES = 0.002 (for the control group at baseline compared to the control group included in the analysis, i.e. those for whom a 
post-test score was established). 
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Outcomes and analysis 


As seen in Figure 1 (above) the number of analysed pupils involved in each school varied between 1 
and 33 with more than half of schools having 8 participating pupils or fewer. Only 14 out of 66 schools 
had 16 or more pupils included in the final analysis. In addition, although participants were recruited 
through schools, parents did not attend a programme according to the school their child attended. As a 
result of the lower number of pupils per school recruited, and the lack of a relationship between the 
group a parent attended and the school their child attended, we did not include school as a factor in the 
model. We initially checked whether there were any interactions between treatment and BPVS (the pre- 
treatment assessment of ability) or with cohort. Having found no evidence that interactions were present, 
the final model included BPVS, gender, FSM, cohort, and treatment with age included for the non-age 
standardised measures—SDQ, PARYC, Year 1 Phonics and Key Stage 1 reading outcomes. 


Given that school was not included in the model, and that the total number of pupils recruited was higher 
than originally estimated, a post-hoc analysis was completed. Optimal Design software was used to 
estimate the statistical power of the analyses using the completed sample size for a single-level trial. 
The calculated pre-post correlation showed a lower than expected correlation between the pre-test and 
the primary outcome (r?= 0.9). Using this revised figure, and 0.05 as the criterion for statistical 
significance, it was determined that a minimum detectable effect size (MDES) of 0.216 could be detected 
with a power of 0.80. 


As specified in the protocol, an intention-to-treat analysis was conducted on all eligible pupils, alongside 
sub-analyses of whether or not the programme was more or less effective when considering only (1) 
FSM pupils (children categorised as ‘ever FSM’ in the NPD), (2) girls, and (3) boys. The protocol also 
specified analysis according to parental level of participation in the programme. Low levels of attendance 
meant that, in discussion with the developers, the two categories of ‘completers’ and ‘non-completers’ 
were created, the former being deemed to have completed the programme if they attended five or more 
of the ten sessions. Consequently, outcomes for the ‘completers’ were compared with those for the 
control condition in this sub-analysis. 


Given that this intervention is aimed at struggling readers, an additional post-hoc sub-analysis was 
conducted for those participants with lower than average attainment on the pre-test. Only children whose 
BPVS score was more than 0.5 standard deviations below 100 (that is, who had received a standardised 
score of 95% or less) were included in this analysis. 


There was evidence that those children lost to follow-up were different from those retained (with 
generally lower baseline scores). Due to the imprecise nature of extrapolating their data they were not 
included in the analysis at subsequent time points. 


As seen above, for the study as a whole there were no statistically significant differences at the 
conventional level at pre-test between treatment and control. In addition, assumptions of normality and 
equal variability held for the primary literacy outcomes and Key Stage 1 reading and reading/writing 
assessment outcomes. For Year 1 Phonics outcomes the residuals (not reported here) tended to be 
negatively skewed. A square transformation was used and results in the assumptions holding although 
it made little difference to the conclusions drawn. Un-standardised means are given in the tables. 


The results of the overall analysis, and sub-analyses, are reported below and in Appendices D-I. Where 
an effect size was found to be statistically significant at the conventional level this is indicated in the 
table by an asterisk (*). 


Intention-to-treat analysis 
Outcomes at post-test 


As seen in Table 9, there were no statistically significant differences between the results for the control 
and intervention conditions at immediate post-test for the primary (literacy) measures for the whole 
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sample (all eligible pupils) participating in the evaluation. While the effect sizes were positive in favour 
of the SPOKES programme for the primary measures they were small (representing approximately one 
months’ progress) and were likely to have occurred by chance rather than as a result of participating in 
the intervention. 


Table 9: Primary outcomes at post-test (all eligible pupils) 


Standardised scores Effect size 


Outcome 
Intervention fefeynlige)| 


N Mean N Mean 
(missing) | (95% Cl) (missing) (95% Cl) 

as 107.95 ts 107.24 620 0.08 
Letter ID >) (107.02, 8.32 or «(108.38 8.07 (291, (-0.07, 
108.87) 108.09) 329) | 0.23) 

7 121.42 oa 119.93 604 0.05 
Word ID oy (119.90, | 13.44 (26) —«(118.54, 12.89 (286, (-0.10, 
122.93) 121.32) 318) 0.20) 

118.98 118.36 604 0.03 
eloeee oD (11799, | 8.77 a (117.41, 888 | (283, (-0.13, 
119.97) 119.32) 321) 0.18) 


In addition, there were no statistically significant differences between conditions for the secondary 
(social and behavioural) measures at immediate post-test (Table 10). The results were also inconsistent 
with both positive and negative effect sizes occurring ranging from -0.08 to +0.18. 
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Table 10: Secondary Outcomes - social and behavioural measures at post-test (all eligible 
pupils) 


St Wy exe) ft] Effect size 
(OVI Kero) qai=) 
Intervention level nl ige)| 
Nin Effect 
SD N Mean SD rrayeye(=1| Size 
(missing) (missing) (95% Cl) (Int, (95% 
Con) Cl) 
SDQ 
7.20 6.93 526 «0.08 
pipsodal on (89, 249 301 664, 253 | (244, (0.09, 
Score (27) 7.50) (19) | 7.22) 282) 0.24) 
Total - 7.36 - 7.70 527. —--0.02 
Difficulties oe (6.68, 561 fa) (7.03, 8.85 (244, (0.19, 
Score 8.04) 8.38) 283) 0.14) 
_ 4.49 4.65 5270.01 
Exemalising:|| 20! (4.02, 3.94 302 (447, 411 | (244, © (0.17, 
Sank (27) 4.97) (19) 5.12) 283) 0.16) 
7 2.86 3.06 527. —--0.04 
internalieng) | 2rd (251, 294 302 (69, | 315 | (244, (0.20, 
rolls, (27) 3.22) (19) | 3.49) 283) 0.13) 
1.13 4.10 228 0.18 
Impact fee (0.88, | 137) ‘134 | ose, | 135 | (103, | (0.06, 
paole ig) 1.38) (9) 1.33) 125) 0.43) 
PARYC 
41.08 41.10 357-002 
Supporting (40.28, 5.21 220 (40.28, 6.09 | (155,  (-0.22, 
Pregnancy (is) 41.88) (18) | 44.93) 202) 0.18) 
1.11 1.14 3570.01 
supporting ie (092, | 1.22 | 729 | oz, | 125 | (155, | (0.21, 
Problematic (15) 1.29) (18) 1.31) 202) 0.19) 
. 38.38 39.48 356 —--0.08 
Lies ees (37.32, 6.90 as (38.54, 693 | (154, (-0.28, 
39.44) 40.41) 202) 0.12) 
. 1.16 4.10 356 ~—- 0.03 
Biecas 169 (092, 153 229 ©0914, 144 | (154, (0.17, 
Problematic (15) 1.40) (18) 1.29) 202) 0.23) 
79.46 80.68 351 0.07 
pec. a (7787, 1026 2% | (7g12, 11.45 | (152,  (-0.27, 
Eneqmeney (19) 81.04) (18) | 99.93) 199) | 0.13) 
2.29 2.26 351 0.01 
Total 167 217 
Problematic (15) Se 2.30 7 (48) = ao ion oo 
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Outcomes at six-month and twelve-month follow-up 


Tables 11 and 12 show that at the six- and twelve-month follow-up there were no statistically significant 
differences between control and intervention conditions for the WRMT-R sub-tests (Letter Identification, 
Word Identification, and Word Attack). The effect sizes do increase between the immediate post-test 
and the six-month follow-up and the twelve-month follow-up although these remain non statistically 
significant and may be a related to the smaller numbers analysed at successive time points. 
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Table 11: Primary outcomes at six-month follow-up (all eligible pupils) 


Standardised scores Effect Size 


Outcome | 


Intervention (Oey ni i Ke) | 


N Mean SD N Mean 
(missing) (95% Cl) (missing) (95% Cl) 

oa 104.17 a 103.07 469 0.15 
Letter ID (35) (103.32, 667 1) (102.11, | 7.80 | 218, (-0.03, 
105.03) 104.03) 250) 0.32) 

Be 120.43 a 118.59 423 0.11 
Word ID (0) (118.62, 13.39 2) (116.81, 14.05 | (197, (-0.07, 
122.24) 120.38) 226) 0.29) 

118.94 117.49 466 0.11 
Mise ae (117.73, 9.30 7 (116.29, 990 (218,  (-0.06, 
120.14) 118.70) 248) 0.28) 


Table 12: Primary outcomes at twelve-month follow-up (all eligible pupils) 


Standardised scores 
Outcome 
Intervention fefeynl ine) | 


N Mean N Mean SD 
(missing) | (95% Cl) (missing) (95% Cl) 

a 97.48 " 95.96 143 0.13 
Letter ID Gs (95.81, | 7.42 a (94.09, 892 (64, (0.17, 
99.15) 97.83) 79) 0.43) 

es 111.45 = 108.45 106 0.27 
Word ID a (107.80, 14.39 (it) (104.71, 15.54 (48, (-0.07, 
115.11) 112.19) 58) 0.62) 

114.64 111.42 140 0.25 
pis ae (114.90, | 12.09 a (108.25, 14.93 (63,  (-0.05, 
117.38) 114.58) 77) 0.56) 


Statistically significant differences at the conventional level and a positive effect in favour of the 
intervention condition were found for two sub-tests of the Strengths and Difficulties Questionnaire (SDQ) 
at the six-month follow-up (Total Difficulties score: Effect Size = 0.217; Internal Difficulties score: Effect 
Size = 0.21; ps0.05). Results for the other sub-tests of the SDQ and the PARYC were not statistically 
significant. The effects do not appear at the twelve-month follow-up point (Tables 13 and 14). 
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Table 13: Secondary outcomes—social and behavioural measures at six-month follow-up (all 
eligible pupils) 


Raw scores Effect Size 
(OV Kee) tai=) 
Intervention fexeynlixe)| 
N in 
Effect 
a el ey yaa Mean sp mosel size (95% 
(missing) (95% Cl) (missing) (95% Cl) (Int, Cl) 
oxey)) 
SDQ 
. 7.35 7.32 359 0.05 
pitas a (6.95, 2.69 ae (6.98, 249 (165, (-0.15, 
7.74) 7.66) 194) 0.25) 
Total ee 8.57 ss 7.48 359 0.22 
Difficulties a (755, | 697| On (6.66, 6.02 (165, (0.02, 
Score 9.59) 8.30) 194) 0.41)* 
- 4.98 4.38 359 0.16 
ibe oe (431, 4.60 ae (3.83, 4.03 (165, — (-0.03, 
5.66) 4.93) 194) 0.36) 
¥ 3.58 3.10 359 0.21 
shel > (3.07, 3.48 ae (266, | 3.23| (165, (0.02, 
4.09) 3.54) 194) 0.41)" 
1.31 1.30 173 0.13 
es a (0.99, 1.57 a (1.03, 1.38 (81, (-0.15, 
1.64) 1.58) 92) 0.41) 
PARYC 
; 42.23 41.62 226 0.09 
a a (41.20, 5.52 a (40.71, 5.29 (101,  — (-0.16, 
el 43.27) 42.52) 125) 0.34) 
0.90 1.05 226 -0.13(- 
Siatulel a (066, 1.27 ee (0.83, 1.28 (101, 0.38, 
1.13) 1.27) 125) 0.12) 
39.76 40.33 224 0.11 
Lesa a (38.61, 6.08 ee (39.31, 5.99 (99,  (-0.36, 
etl 40.91) 41.35) 125) 0.14) 
; 0.92 0.99 224 -0.07 
Rae ne (0.66, 1.36 ao (0.72, (157, (99, | (031, 
1.18) 1.25) 125) 0.19) 
81.86 81.98 220 0.04 
Rien ae (79.95, 10.04 a (80.28, 9.88 (97, (-0.29, 
amid 83.78) 83.67) 123) 0.21) 
1.80 2.04 220 -0.10 
Total 110 136 
. (1.38, 2.20 (1.62, |250| @7, | (036, 
Problematic (13) 2.22) (13) 2.47) 123) 0.15) 
*p<0.05 
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Table 14: Secondary outcomes—social and behavioural measures at twelve-month follow-up (all 
eligible pupils) 


Raw scores Effect Size 
OTE] Kero} ual: 
Intervention Control 
Nin 
| Mean N Mean SD model _ Effect Size 
(missing) (95% Cl) (missing) (95% Cl) BG (95% Cl) 
on 
SDQ 
: 7.44 7.16 123 0.06 
Prosocial if (6.76, 2.83 i (6.61, 238 (57, (-0.27, 
Score (14) 8.11) (7) 7.72) 66) 0.38) 
Total 74 8.54 73 8.33 123 0.14 
Difficulties (ia) (6.79, | 7.35 a) (6.94, 594 (57, (-0.19, 
Score 10.28) 9.72) 66) 0.46) 
= 5.10 5.08 123 0.11 
Externalising 71 (3.92, 4.98 73 (4.10, 418 (57, (-0.22, 
Score (14) 6.28) (7) 6.06) 66) 0.44) 
4 3.44 3.25 123 0.12 
Internalising 71 (2.59, 3.55 73 (2.54, 3.00 (57, (-0.21, 
Score (14) 4.28) (7) 3.95) 66) 0.44) 
1.62 1.32 64 0.27 
impact = (1.02, 1.94 - (0.77, 163 (34, (0.18, 
Score (8) 2.22) (4) 1.88) 30) 0.73) 
PARYC 
: 42.95 41.03 59 0.24 
Supporting 38 (41.07, 5.78 Fo (38.83, 6.33 (31, (-0.23, 
Frequency (7) 44.82) (5) 43.23) 28) 0.71) 
; 0.89 1.33 59 -0.26 
Supporting 38 (0.46, 1.33 ae (0.78, 159 (31, (-0.73, 
Problematic (7) 1.33) (5) 1.89) 28) 0.21) 
. 41.16 39.52 59 0.29 
Proactive 38 (38.87, 7.06 33 (37.45, 5.92 (31, — (-0.18, 
Frequency (7) 43.45) (5) 41.58) 28) 0.75) 
: 0.71 1.09 59 -0.26 
Proactive 38 (0.16, 1.69 = (0.55, 155 (31, (-0.73, 
Problematic (7) 1.26) (5) 1.63) 28) 0.21) 
84.11 80.55 59 0.29 
Total 3880.39, 11448 (76.65, 11.19 (31, (0.18, 
Frequency (7) 87.82) (5) 84.44) 28) 0.76) 
1.61 2.42 59 -0.28 
Total 38 33 
Problematic —(7) ees - (5) ois ail oa) O19)" 
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Year 1 Phonics results 


As seen in Table 15, there were no statistically significant differences at the conventional level seen on 
the results for the Year 1 phonics assessment for cohorts one to five. 


Table 15: Year 1 Phonics results (all eligible pupils) 


Sy rTavek-TKe | 1-10 Mtere) «=1-4 Effect Size 
Outcome 
Intervention (Oey ai ime) | 

N in 

N Mean SD N Mean rayeye(:)| ees 

(missing) (95% Cl) (missing) (95% Cl) (Int, a 
(95% Cl) 

efoJ)) 

30.02 29.93 524 0.02 

se ae (28.83, | 9.99 es (28.72, | 10.37 | (252, | (0.15, 

31.20) 31.14) 272) 0.18) 


Secondary analysis: Key Stage 1 assessment literacy results 


Although not specified in the protocol, statistically significant effect sizes were seen for the KS1 reading 
scores (p = 0.01) and the reading and writing scores combined (p = 0.049) for those pupils for whom 
these results were available (cohorts one and two), representing three to four months progress in favour 
of the intervention condition (Table 16). 


Because Key Stage results are an outcome of such interest to schools, we felt it was valid to include 
these results here as secondary analysis. It is important to note that the analysis is for two out of the six 
cohorts only (those for which KS1 data was available) and so does not give the full picture of the impact 
of the intervention on KS1 results. 


Table 16: Key Stage 1 assessment results (all eligible pupils) 


ST FT ale F-1K0 | (x0 oxexe) g-4-) Effect Size 
Outcome 
Intervention fefelnlixe)| 
N in 
N i Kexetal SD N i Kexetal D arexe(=y| os 
Smee < ee 
(missing) (95% Cl) (missing) (95% Cl) eae (95% Cl) 
KS1 
Assessment 114 13:99 111 1508 3.5 189 0.26 
Reading/Writi (22) (13.24, 3.69 (14) (12.41, A (92, (0.00, 
ng Score 14.62) 13.75) 97) 0.52) 
KS1 
Assessment 114 nied 111 Tee" ag). Ie 0.33 
Reading (22) (13.92, 3.9 (14) (12.87, 5 (92, (0.07, 
She 15.41) 14.28) 97) 0.59)* 
*p<0.05 
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Sub-analysis 
Pupils in receipt of Free School Meals 


As seen in Tables 17-24 there were no statistically significant differences found between control and 
intervention conditions for children in receipt of Free School Meals for the primary or the secondary 
outcomes at immediate post-test, six-month follow-up or at twelve-month follow-up. 


Outcomes at post-test 


Table 17: Primary outcomes at post-test (all eligible FSM pupils) 


Standardised scores Effect Size 
Outcome 
Intervention | Control 
Nin Effect 
ye laa N De model _ Size 
(missing) (95% (missing) (95% 12) (Int, (95% 
a) ) Con) Cl) 
88 105.72 99 106.38 171 0.02 
Letter ID (8) (103.72, 9.35 (a) (104.58, 9.10 (80, (-0.27, 
107.71) 108.21) 91) 0.31) 
86 116.76 96 117.46 168 -0.00 
Word ID (113.64, | 14.43 (114.55, | 14.25 (79, (-0.29, 
(7) (7) 
119.87) 420.37) 89) 0.29) 
117.14 116.87 164 0.09 
pies a (115.16, 9.07 mn (11489, 968 § (77, | (-0.21, 
119.12) 118.86) 87) | 0.38) 
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Table 18: Secondary Outcomes - social and behavioural measures at post-test (all eligible FSM 
pupils) 


Raw scores Effect Size 
(OVI Kexe) pai=) 
Intervention fefeynlige)| 
N Mean N Mean 
de (95% a (95% 
(missing) Cl) (Crest ale p) Cl) 
SDQ 
6.63 7.01 147-015 
bi si o (6.06, 2.54 . (6.48, 244 (70, | (-0.46, 
7.21) 7.55) 77) 0.16) 
Total - 9.46 a 9.52 1470.00 
Difficulties is (8.09, 6.09 «40, 8.07 (70, (-0.30, 
Score 10.83) 10.63) 77) 0.31) 
¥ 5.97 5.64 147 (0.08 
oe i. (4.96, 4.51 a (4.73, 412 | (70, (-0.23, 
6.99) 6.54) 77) 0.39) 
- 3.48 3.88 1470.11 
oe os (2.80, 3.03 . (3.20, 3.41 | (70, (-0.41, 
4.16) 4.56) 77) 0.20) 
4.12 1.02 on 0.22 
(lla o (0.73, 1.39 S (0.70, 112 (45, | (-0.17, 
1.51) 1.34) 46) 0.62) 
PARYC 
; 40.70 40.60 78 --0.08 
Selatan . (38.80, 5.46 . (38.65, 7.12 | (30, (-0.51, 
ee 42.60) 42.56) 48) 0.36) 
1.27 1.23 78 0.01 
Supporting oS (0.81, 1.33 ae (0.88, 127 (30, — (-0.43, 
Problematic (3) 1.74) (5) 1.57) 48) 0.44) 
38.03 38.53 81 0.15 
seid a (35.67, 6.87 55 (36.35, 8.09 ~~ (31,_~—Ss«(-0.28, 
Frequency (3) (5) 
40.39) 40.71) 50) 0.58) 
; 1.74 1.18 81 0.16 
Proactive oe (1.19, 1.58 a (0.78, 148 | (31, (-0.27, 
Problematic (3) 2.28) 9) 1.58) 50) 0.59) 
78.70 79.17 78 (0.05 
etal oS (74.97, 10.71 53 | (75.47,| 13.47 | (30, | (-0.39, 
Frequency (3) (5) 
82.43) 82.87) 48) 0.49) 
3.06 2.43 78 0.07 
Jetall, ae (2.34, 2.08 oe (1.81, 228 (30, | (-0.37, 
pace leinalie (3) 3.78) (5) 3.06) 48) 0.50) 
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Outcomes at six-month and twelve-month follow-up 
Table 19: Primary outcomes at six months (all eligible FSM pupils) 


SS} F-Taver-1Ke | f-y-10 Myere) «-1-y Effect Size 
LO]T] Korey nay=) 


Intervention | Control 


Mean Mean 


heer a N 9 
N (missing) ae (missing) a 


i 103.55 ¥ 101.35 129 | 0.30 
Letter ID . (101.83, 6.98 «(99-29 8.93 (60, (-0.04, 
105.26) 103.41) 69) 0.63) 

Bs 117.81 a 116.03 190.11 
Word ID . (113.96, 15.26 @) (112.48, 14.74 (57, (0.24, 
121.65) 119.60) 62) 0.45) 

118.31 115.87 128 0.15 
bane i. (115.71, 10.49 ic (113.37, 10.82 (59, (-0.18, 
120.91) 118.37) 69) 0.48) 
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Table 20: Secondary outcomes—social and behavioural measures at six-month follow-up (all 
eligible FSM pupils) 


Raw scores Effect Size 
(OlVi Kero) taK=) 

Intervention level nlixe)| 

N Mean N Mean 

An (95% oh (95% 

(missing) Cl) (exitare) Cl) 

SDQ 
. 6.58 7.39 102. -0.15 
pitas ) (587, 2.67 . (6.73, | 253 | (49, | (-051, 
7.29) 8.05) 53) 0.22) 
Total - 10.67 ” 9.19 1020.19 
Difficulties a (8.74, 7.27 | «(775 | 8.50 (49, | (-0.18, 
Score 12.59) 10.62) 53) 0.55) 
- 6.89 5.32 102 0.26 
ibe . (554, 5.11 . (4.30, 3.91 | (49, | (-0.10, 
8.25) 6.34) 53) | 0.63) 
¥ 3.77 3.86 102 | 0.01 
shel . (288, 3.35 . (3.00, 3.32 (49, (0.36, 
4.66) 4.73) 53) 0.37) 
4.21 1.18 69 ~—s0.20 
es a (069, 1.60 S (0.78, 124 (32,  (-0.25, 
1.73) 1.57) 37) 0.64) 
PARYC 

; 44.04 40.34 52 0.67 
a . (42.13, 4.86 . (38.33, 5.70 | (23, (0.14, 
el 45.95) 42.36) 29) —-1.20)* 
0.92 1.31 52 0.09 
Supporting 26 (047, 1.16 Be (084, 133 (23,  (-060, 
pleblomiaie (3) 1.38) (3) 1.78) 29) 0.43) 
39.38 37.91 530.03 
Lesa x (36.66, 6.94 . (3583, 596 (23,  (-0.54, 
BE nRY 42.11) 39.98) 30) 0.49) 
1.58 4.12 53 | 0.30 
pioaeve = (0.91, 1.70 oe (0.53, 169 | (23, (-0.21, 
Problemane (3) 2.24) (3) 1.71) 30) 0.82) 
83.42 78.13 52 0.34 
Rien S (79.68, 9.54 . (74.59, 9.99 (23,  (-0.18, 
ae 87.16) 81.66) 29) 0.86) 
2.50 2.44 52 0.17 
Total oe (1.55, 2.42 ae (1.51, 263 | (23,  (-0.35, 
paeniemate (3) 3.45) (3) 3.37) 29) 0.69) 


*p <0.05 
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Table 21: Primary outcomes at twelve-month follow-up (all eligible FSM pupils) 


Standardised scores Effect Size 
Outcome 
Intervention | Control 
Mean N Mean 
N (missing) (95% (missing) (95% 
for) g Cl) 
32 97.09 31 96.84 54 0.10 
Letter ID (4) (94.54, 7.22 6) (93.74, 863 (28, — (-0.39, 
99.65) 99.94) 26) 0.60) 
4 107.48 24 105.50 38 0.21 
Word ID (3) (101.14, 14.51 (4) (99.99, 13.51 (18, (-0.38, 
113.81) 111.01) 20) | 0.80) 
113.91 110.20 53 0.23 
bane a (109.70, 11.90 a (105.02, 14.18 (28, — (-0.27, 
118.11) 115.38) 25) 0.73) 
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Table 22: Secondary outcomes—social and behavioural measures at twelve-month follow-up (all 
eligible FSM pupils) 


Raw scores Effect Size 
(OlVi Kero) taK=) 
Intervention fefelnlixe)| 
WY Kee Ta) Mean 
0 0 
(missing) wie sp (missing) va 
sDQ 
. 7.26 7.32 46 0.01 
pitas re (6.28, 2.55 S (6.51, 201 (24, | (-0.55, 
8.24) 8.13) 22) 0.53) 
Total - 10.07 - 9.48 46 0.12 
Difficulties S (7.50, 6.68 a (7.26,| 555 | (24, | (-0.42, 
Score 12.65) 11.70) 22) 0.67) 
7 6.63 5.80 46 0.24 
ibe ze (4.75, 4.88 a (4.14, 414 © (24, (-0.31, 
8.51) 7.46) 22) 0.78) 
. 3.44 3.68 46 -0.10 
wee . (2.26, | 3.07 . (2.43, 312 (24, | (065, 
4.63) 4.93) 22) 0.44) 
1.39 4.21 29 0.00 
es ns (0.68, 1.50 a (0.42, 148 (16, (-0.70, 
2.10) 2.00) 13) 0.70) 
PARYC 
; 42.00 41.75 18 --0.17 
a a (37.50, 7.79 a (37.90, 544 (11, | (-1.07, 
qneney 46.50) 45.60) 7) 0.73) 
0.67 1.25 18 0.16 
Supporting ie (0.10, 0.98 : (0.52, 104 (11, | (-0.73, 
Problematic (1) 1.24) (1) 1.98) 7) 1.06) 
40.42 37.75 18 0.14 
Lesa o (35.69, 8.18 (32.16, 7.91 | (11, (-0.76, 
seeded 45.14) 43.34) 7) 1.04) 
; 0.83 1.13 18 0.23 
Proactive Oa (0.08, 1.59 2 (0.09, 146 | (11, (-0.67, 
Problematic (1) 1.75) (1) 2.16) 7) 1.12) 
82.42 79.50 18 -0.00 
Boley a (73.84, 14.85 - (70.64, 1252 (11, | (-0.90, 
90.99) 88,36) 7) 0.89) 
1.50 2.38 18 0.22 
Total 12 8 
. 0.08, 2.47 0.87, 2.13 11, (0.68, 
Problematic (1) Ses (1) ab . ere 
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Year 1 Phonics results 


Table 23: Year 1 Phonics results (all eligible FSM pupils) 


Standardised scores Effect Size 
Outcome 


Intervention | forey nace) 


Mean 
(95% 
(e3)) 


N 
(missing) 


Year 1 82 27.27 ag 27.53 151 -0.11 
Phonics (11) (24.55, 12.30 (7) (25.08, 11.42 (71, (-0.41, 
29.99) 29.98) 80) 0.20) 


Key Stage 1 assessment literacy results 


Table 24: Key Stage 1 assessment results (all eligible FSM pupils) 


Sy e-Tavek-1ue | 1-10 M-yere) a=1-) Effect Size 
Outcome 
Intervention | Control 


KS1 
13.28 12.47 76 0.10 
“Reading! a) (1220, 3.0 C (11.49, 3.21 | (39, (-0.31, 
Writing Score 136) 13.44) 37) 0.52) 
KS1 
13.87 12.86 76 0.17 
oo . (12.72, 3.90 é (11.84, 3.34 (39, (-0.25, 
fs g 15.02) 13.88) 37) 0.59) 
core 
Gender 


No differences were found for the primary and secondary outcomes at immediate post-test. Statistically 
significant differences at the conventional level were, however, found between the results of boys at six- 
and twelve-month follow-up, although follow-up results were not available for all boys in the trial. At six 
months, boys in the intervention condition demonstrated positive effects representing approximately 
three months’ progress in Word Identification and Word Attack compared to those in the control 
condition (ES = 0.25; p = 0.04 and ES = 0.24; p = 0.05, respectively). At twelve-month follow-up, results 
representing approximately seven months’ progress were seen for Word Attack (ES = 0.53, p = 0.02). 
In addition, although the KS1 SATs outcomes were not included in the protocol and results were only 
available for cohorts one and two, positive effects were also found for KS1 reading outcomes (ES = 
0.41; p = 0.02), representing approximately five months’ progress in favour of the intervention. There 
were no statistically significant differences found in any of the measures at any time point for girls. The 
results of this analysis are presented in Appendix D. 
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Low achieving children at baseline (score of 95% or less at baseline) 


There were no statistically significant differences in primary and secondary (social and behavioural) 
outcomes at any time point except for the ‘Supporting Frequency’ (supporting positive behaviour) 
measure at six-month follow-up where an effect size of 0.59 was seen (p = 0.03). The results of this 
analysis are presented in Appendix E. 


On-treatment outcomes: parental level of participation in the SPOKES programme 


The SPOKES developers specified that attendance at five or more sessions of the programme would 
be regarded as ‘completing’ the programme. Using this measure, no statistically significant differences 
in primary literacy outcomes, or the Year 1 phonics assessment results, were found between those who 
completed the programme and the control condition. There were, however, positive, statistically 
significant differences in the non-pre-specified Key Stage 1 reading (p = 0.01) and reading/writing (p = 
0.03) measures with corresponding effect sizes of 0.42 and 0.35 representing approximately five and 
four months’ progress for those receiving the SPOKES programme respectively. Results are presented 
in Appendix F. 


Cost 


The costs for SPOKES are somewhat different than for most of the EEF projects in that the intervention 
is delivered to the parents, not the children, thus there may not be a direct cost to the school. The 
programme could be funded by a school, or it could be funded by a social service organisation. 


Details of the costs of implementing the SPOKES programme were obtained from Plymouth Parent 
Partnership. These costs are primarily staff costs to implement the weekly SPOKES sessions. The cost 
for the programme implementation in this study was approximately £804 per place, which could be 
considered per child, per parent, or per family. A breakdown of the total costs can be seen in Table 25. 


Table 25: Breakdown of costs for programme delivery (3 cohorts) 


Education psychologists (1 day per week) £68,040 
Parenting Programme Facilitators (1.5 days per week) £60,312 
Training £5,200 
Administration £10,145 
Supervision £4,500 
Childcare (créches for families) £7,251 
Books £7,085 
Resources £2,250 
Venues* £8,921 
Total £173,704 


* Not all venues charged for room use. Cost per room was approximately £450 per group. 


These costs were calculated over three rounds of the programme (18 courses across the three 
cohorts—a total of 216 parental places). The total costs for these were £173,704. 
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This was an efficacy trial, designed to be delivered under ideal conditions with additional support by the 
developer. A small proportion of these costs, therefore, would not apply to a normal implementation of 
a scaled-up programme. For example, the training costs would be reduced over time once group leaders 
were already trained; in addition, if sessions were held in schools free-of-charge, venue costs could be 
reduced making the average cost per place £794. However, this is still a relatively expensive 
intervention, albeit comparable in cost to other parent programmes. 


Education Endowment Foundation 38 


SPOKES 


Process evaluation 


This section summarises the key findings from the process evaluation. It includes attendance data 
collected by group leaders, an overview of the responses of the subgroup of intervention parents who 
took part in the telephone interviews, summaries of the evaluations of SPOKES that parents completed 
at the end of the programme, and a sub-analysis of the weekly evaluation returns. 


Implementation 


As this evaluation of SPOKES was an efficacy trial—a trial of the programme run in optimum 
conditions—high standards of implementation were required to ensure the programme was delivered 
as it was designed. Implementation fidelity was ensured by the developers and the Plymouth Parent 
Partnership through weekly supervision meetings with the developers and group leaders. These 
supervision sessions involved detailed observations of sessions recorded by group leaders, and 
interrogation of specific activities, with an emphasis on fidelity to the manual. 


An important factor in terms of implementation fidelity is the dosage—in this case, attendance at 
parenting sessions. As the programme was not based around a particular school or location, strategies 
were put in place to boost attendance, for example through providing taxis for those who would 
otherwise struggle to attend and providing, where possible, city centre locations for sessions. In addition, 
a requirement for eligibility for inclusion in the trial was the ability to attend sessions if allocated to the 
intervention condition. 


Parental attendance 


This section includes a summary of the number of sessions that parents attended, and the main reasons 
provided for any absences. 


The intervention involved ten sessions with parents. As indicated in Table 26, attendance at SPOKES 
sessions was lower than anticipated. Over a quarter (27%) of parents assigned to the programme did 
not attend any sessions. Only 7% attended all sessions. As noted, attending five sessions was 
considered ‘completion’: just over half of participants (57%) who were allocated to the intervention 
achieved this. This remains relatively consistent across cohorts. 
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Table 26: Attendance at SPOKES sessions per cohort as reported by the Plymouth Parent 
Partnership 


Cohen | C1 C2 Total 


Tan) eX) me) mel -¥-t-J (0) aT Ua ccyare [=o] | 


0 17 15 19 18 13 18 100 

1 5 3 2 5 3 rg 25 

2 0 5 0 rs 2 4 13 

3 1 1 2 2 2 2 10 

4 1 2 3 2 1 2 11 

5 2 1 5 4 2 3 17 

6 2 5 3 2 4 5 21 

t 4 7 4 3 8 5 31 

8 id 9 8 13 9 10 56 

9 11 10 5 10 11 14 61 

10 0 7 3 8 2 5 25 

Completers (attended 5+ sessions) | 

Number 26 39 28 40 36 42 211 
Percentage (%) 52 60 52 58 63 56 57 


* Excludes ineligible participants and those who withdrew consent to use data. 
End of programme and weekly evaluation forms 


One hundred and seventy-three end of programme evaluation forms were completed by participants. 
The original evaluation form was completed by 49 parents in the intervention condition in cohorts one 
and two. The form was then altered to make the questions more valuable for evaluation purposes. The 
revised evaluation form is given in Appendix G, along with the numbers and percentages for each Likert- 
scale response. This was completed by 124 parents in cohorts three to six. 


Overall, it can be seen that parents were very happy with the programme, with the vast majority finding 
aspects of the SPOKES techniques and content ‘useful’ or ‘very useful’, the methods used in group 
sessions ‘useful’ or ‘very useful’, and group leaders ‘helpful’ or ‘very helpful’. 


Three to four per cent of respondents found ‘using environmental print’, ‘visiting the library’, and the 
‘one-to-one visit’ ‘not very useful’ or ‘not at all useful’ (although only four parents per item expressed 
these views). All of the weekly evaluation forms were, however, filled in by intervention parents regarded 
to have ‘completed’ the programme. 


The weekly evaluation forms from sessions one and two were analysed to see if there were any 
differences between the responses of ‘completers’ and ‘non-completers’. These weeks were chosen 
because they were the only sessions for which there were regarded to be sufficient numbers of ‘non- 


Education Endowment Foundation 40 


SPOKES 


completers’ returning forms. The weeks were analysed separately because each session was on a 
different topic and this would ensure comparability between the two groups. A copy of the feedback form 
is provided in Appendix H. Two-hundred and forty-one respondents completed the feedback forms for 
session one (49 non-completers and 190 completers), and 185 for session two (20 non-completers, 165 
completers). 


Analysis of the main questions regarding enjoyment of the session and its usefulness indicated that in 
both sessions the mean response of completers was higher (and therefore more positive) than that of 
non-completers (four and three respectively for both questions in session one; and five and four 
respectively for both questions in session two). Given the small numbers and diversity of individual 
responses, however, it is difficult to explain the reasons for these differences, although there is some 
indication that a combination of dislike of role-play, personal circumstances (including personal 
shyness), and the conditions of the session (such as room size and the presence of other parents) may 
have been factors. For example, free responses to the open-ended question ‘What did you like least 
about today’s session?’ for the week one evaluation form indicate that 15% of responses to this question 
by completers named the role-play element of the session (17 out of 107 responses) compared to 13% 
of non-completers (3 out of 23 responses), 21% of non-completers indicated personal circumstances 
(including shyness) compared to 5% of completers and 25% and 26% of completers and non-completers 
respectively mentioned the conditions of the sessions. In contrast 40% of completers compared to 25% 
of non-completers indicated ‘nothing’ in response to this question. 


Parent interviews 


Evaluation team staff called 130 parents from the intervention group to participate in brief interviews at 
each post-test time period. Interviewers attempted to reach parents up to four times but only reached 
59 parents. Reasons for inability to contact remaining parents included having the wrong number, 
getting no answer, or the line being always busy. 


Attendance from parent interviews 


In contrast to recorded attendance by group leaders, in the telephone interviews the majority of parents 
reported missing very few sessions, and 80% of respondents (56 out of 71) reported that they attended 
five or more sessions—somewhat higher than the attendance recorded by the group leaders for the 
whole treatment group where only 57% were recorded as ‘completing’ the programme using this 
definition. This was to be expected as those who attended more often were more likely to participate in 
the telephone call. In addition, there is likely to be a relationship between attitudes towards the 
programme and attendance. 


Although their attendance levels were high, the telephone interviewees did provide some reasons for 
non-attendance at sessions. The principle reasons given for missing sessions were their own or family 
illnesses, work, and holidays. Some parents felt that the initial invitation letter and consent forms were 
unclear about the requirements of their participation. When asked what would make it easier to attend 
a few cited the location and the timing of the sessions. 


Of the 12 parents in the telephone interviews who reported low or non-attendance (for three interviewees 
attendance information provided was incomplete), five reported this as being due to their own or a child’s 
illness, and three reported changes in employment (including study) and/or workload. Three parents 
also reported issues with caring for other children, despite provision being made for childcare during 
attendance at the programme to improve accessibility. It should be noted, however, that two of these 
parents also reported illness and starting studying as other life changes impacting on their ability to 
attend the programme. Of the three remaining parents, one missed the first two weeks and then felt she 
couldn’t attend because she ‘may have missed too much’. Two, however, felt that they did not think their 
child needed any extra support so did not attend: ‘my child is not a struggling reader’. This might indicate 
that some schools nominated children who did not meet the specified criteria or used different criteria 
for recruitment to the programme. 
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Finally, due to the research design, there was a considerable gap between consent, randomisation, and 
the start date for the parenting sessions. When family illness or a house-move cropped up, families 
could not defer a term because of possible cohort effects. So, means of coping with changes in family 
circumstances that could be dealt with in a rolled out version of the programme were not possible in the 
study meaning these families were lost to the evaluation. 


Use of SPOKES strategies or activities 


During the interviews, parents reported using many of the strategies taught in the SPOKES programme. 
By far the most frequently mentioned was the use of Pause, Prompt and Praise. This must have been 
taught, and repeated, frequently as many parents used this phrase to report that they had been praising 
their children, and pausing and encouraging them to read on when they were stuck. Many of them cited 
literacy tips that they used when reading with their children, playing with their children, and ignoring their 
bad behaviour indicating that they used them frequently, and not only with their Year 1 ‘struggling 
reader’. One parent, for example, reported: ‘I use [the techniques] with all our children during family time 
or homework time’. 


When asked what strategies they did not use, there were a number of responses, with the two most 
frequently cited being working with letters and puppets. 


Overall parental satisfaction with the programme 


In the immediate post-programme interviews, parents were overwhelmingly positive about SPOKES. 
Fifty-six of the 59 parents interviewed responded enthusiastically about it. They felt that they, their 
children, and their families had benefitted from their participation: ‘It made me feel like we were all in it 
together’; ‘My son made big steps after this [the programme]’. Of particular importance was being able 
to use the ideas at home: ‘[I enjoyed] putting into place the ideas which [the group leaders] gave to us. 
It was nice to take them home and watch them work. | always looked forward to trying out what they 
suggested’. Most parents would recommend it to other families. Of the three who had reservations, two 
felt it would be better for younger children and one said that her daughter didn’t need it: ‘[I] didn’t dislike 
it but ... [I] felt | didn’t need to be there’. 


In the parent interviews, parents were asked what they liked most about the SPOKES sessions. 
Responses were very positive and varied. The most often mentioned activities were group support, 
learning the Pause, Prompt and Praise strategies, behaviour management techniques, making puppets, 
and sharing ideas. 


There were not many aspects of the programme that parents reported disliking. Of those that were 
reported, role-play within sessions was the most frequently given answer with 16 parents disliking this 
aspect, primarily because they were shy or lacked confidence. 


Follow-up interviews with parents 


The follow-up parent interviews which took place at six months (41 interviews, cohorts one to five) and 
12 months after completion of the programme (nine interviews, cohorts one and two) revealed similar 
patterns of findings among the cohorts and across the two time periods regarding programme 
satisfaction and strategies used. However, because these took place some time after the end of the 
programme, the comments were somewhat more general than in the immediate post-test interviews. 


Their overall reaction to the programme remained positive: parents really enjoyed it, reported learning 
a lot, and said they would recommend it to other parents. The activities that were reported most often 
as still being used by parents were Pause, Prompt, Praise, reading together, and increased wait time 
for the children to read words. At the six-month follow-up interview a few parents remembered not liking 
the role-plays, or sharing sessions with others, because they had felt self-conscious and shy. 
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Suggestions from parents 


From the parent interviews, there were a number of suggestions about what to change about the 
programme but there was no consistency in those suggestions. Some thought SPOKES should be 
longer, some shorter; a few wanted more focus on literacy and less on parenting, and a few suggested 
that it would be more appropriate for younger children (such as those in reception or nursery). A few 
parents reported that the initial letter that they had received about the programme was not clear meaning 
that they did not really understand what they were committing to: ‘[The] letter that went out initially... 
wasn’t clear what it was going to be about. A lot of parents didn’t turn up because they thought it was 
just about buying books’ and that messages from schools could differ: ‘[I] was told by the school that 
the course was only offered to kids with difficulties but then on the course no other parent was told this. 
They were just offered the course’. 
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Conclusion 


Discussion 


The Teaching and Learning Toolkit reports that overall parental involvement in children’s schooling 
results in ‘moderate impact for moderate cost, based on moderate evidence’. Two recent meta-analyses 
from the USA suggested that increasing parental involvement in primary and secondary schools had, 
on average, two to three months’ positive impact (Jeynes, 2005; Nye, Schwartz, and Turner, 2006). 
These findings cover a wide variety of interventions at different age levels and for different subjects, 
some for regular students, and some for those struggling. The main results of this study found weaker 
effects than the previous studies quoted. In addition, previous research on a similar SPOKES 
programme (which had a greater focus on social behaviour) found it to be effective in improving both 
the literacy and behaviour of five- and six-year-old children at risk of social exclusion through antisocial 
behaviour (Sylva et al, 2008) with an effect size (d) of 0.3 for the British Ability Scales word reading sub- 
test (p. 447). 


The overall intention-to-treat findings from this efficacy trial of SPOKES indicate that the programme did 
not have an impact on children’s literacy. The only outcome measures that showed a statistically 
significant effect in favour of the treatment group were the KS1 reading and writing measures. These 
measures were not pre-specified in the trial protocol and the analysis was only able to be conducted on 
a sub-sample of participants (cohorts one and two). As the KS1 assessments were administered and 
marked by the children’s teachers, they are less reliable than the standardised reading measures or 
than the end of Year 1 phonics assessment, both of which are more objective measures. This would be 
true across both control and intervention groups which somewhat increases the validity of these findings, 
although in some cases teachers may have been aware of parental attendance at SPOKES sessions. 
The primary outcomes measures (WRMT-R sub-tests) can be considered more reliable than the KS1 
results and other teacher-completed or parent-completed measures (such as the SDQ and the PARYC) 
as the individual assessors were blind to condition. 


There were, however, some positive findings in the subgroup analysis relating to the impact of SPOKES 
on boys, particularly in the longer term. There were statistically significant findings at the conventional 
level for some, although not all, primary outcomes at both the six-month and twelve-month follow-up 
rounds of data collection, and in the Key Stage 1 reading assessments. Although not all boys were 
included in these analyses, the findings suggest an impact of the SPOKES programme on some aspects 
of boys’ literacy of between three and seven months’ additional progress. 


The secondary, social, and behavioural outcomes in this evaluation were inconsistent. However these, 
and the baseline measure, can be considered to be within the normal range which may suggest that 
enacting substantial improvement may be more difficult than, for example, in the previous SPOKES 
trials that targeted children with behaviour and literacy difficulties. 


The process evaluation findings demonstrated that the programme was positively perceived by the 
parents who attended sessions, that they enjoyed their participation and, in fact, believed that it had 
improved their children’s literacy. Although there are indications that parents introduced techniques 
learnt through the SPOKES programme in the home (such as Pause, Prompt, Praise) this is only 
through self-report and does not indicate the extent of any changes which occurred. The telephone 
interviews indicated that for the majority of those who were low- or non-attenders this was due to 
primarily unforeseen life factors, including illness and employment changes. Yet this was not a 
representative sample: in total only 57% of the parents participated in five or more of the ten parenting 
sessions—not a very high participation rate. This might account for the limited impact on children’s 
reading. Attendance at other parenting programmes has varied across studies but the retention rates in 
this evaluation do appear to be lower than other parent programmes, and in other studies higher 
attendance was correlated with positive effects on parenting (Reid et a/., 2004) although this is not 
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evident in this evaluation. This may, however, reflect the very low criteria for being considered a 
‘completer-—attending only five of the ten sessions. 


Limitations 


There were, however, some limitations to the evaluation which may have impacted on the outcomes. 
These were primarily related to recruitment to the programme, a delay between recruitment and 
implementation, and low attendance. 


The identification by schools of struggling readers does appear to have been problematic, with the 
majority of children not being defined as such by the baseline measure (or, in some cases, by their 
parent), but rather by the children’s teacher (as discussed in the ‘eligibility’ section above). Because 
only a low proportion of children scored poorly on the standardised baseline measure (for this evaluation 
a ‘poor’ score was a score of 95% or less of the mean at baseline, which was the case for 28% of the 
total sample for analysis), the programme can be said to have not been primarily delivered to its intended 
target population. This could have reduced any impacts, both overall and on those children who were 
struggling readers (because the group-based nature of the programme would have meant that parents 
would not have the same shared experience to draw upon, for example in group discussions and role- 
play activities). In addition, between the baseline pre-test measure and the post-test, those who were 
lost from the control condition had, on average, lower baseline attainment than those retained in the 
study, and those pupils lost from the intervention condition with higher baseline attainment were lost to 
follow up. Within this context it should also be noted that children in the control condition did receive 
free books and newsletters in order to encourage recruitment and retention to the study. 


The early recruitment of parents to the programme in order to facilitate the evaluation (in particular to 
allow for baseline measures to be collected) meant that there may have been a long time lag between 
recruitment and the start of the programme for some parents. Consequently, when there was a change 
in circumstances, such as family illness, house-move, or change in employment circumstances—more 
likely to happen over a longer time period—families were lost to the evaluation as the research design 
meant that families could not defer a term because of possible cohort effects. Therefore, ways of coping 
with changes in family circumstances that could be incorporated into a rolled out version of the 
programme were not possible in the study, meaning these families were lost to the evaluation. This 
impacted on recruitment and may also have impacted on the profile of those recruited (such as in favour 
of less struggling readers). Also, parents did not necessarily attend the programme close to their own 
locality or with other parents from their child’s school. As indicated above, attendance on the programme 
was low and it could be queried if the definition of ‘completer’ as having attended five or more sessions 
could be considered sufficient to have had programme impacts. Consequently the programme could be 
considered not to have been delivered to the optimal design desired in an efficacy trial. 


As differences between schools can be large, not taking school into account in the analyses could mean 
that the effect of treatment appeared to be a lot smaller, and so potentially not statistically significant, 
than would have been the case in a clustered analysis. However, we chose not to conduct a clustered 
analysis due to the small number of pupils recruited from many of the schools, and because group 
attendance for intervention parents was not linked to the school their child attended. Conversely, 
although smaller than anticipated numbers were generally recruited from each school, the recruitment 
process was the same for all parents, which may have led to some contamination effects (for example, 
parents allocated to different conditions may have discussed the programme and received the same 
information regarding the programme at the ‘coffee morning’ events). In the analyses all statistically 
significant differences were in favour of the intervention, yet because so many statistical analyses were 
conducted, the likelihood of finding some statistically significant differences by chance increases. 
Importantly, while the baseline and outcome measures used were all designed to assess literacy and 
behavioural outcomes in the target population relevant to this study, the correlation between the pre- 
test and the primary outcome was lower than expected in this study. 
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Further limitations to the design which could have impacted on outcomes include a possible cohort effect 
and the mean delay in some post-testing resulting from the delivery model of the programme. Given 
that the programme was delivered across the school year, the term in which it was delivered could have 
had an impact on outcomes. In other words, there could be a time in the school year during which the 
programme could have higher impact on student outcomes than at other stages. The research design 
also meant that some delay in post-testing did occur (for example, for those children whose parents 
participated in the programme during the Summer term post-testing was delayed by the long summer 
break). In the analysis, the impact of cohort on treatment effect was, however, inconsistent and therefore 
excluded from the model. However, one cohort (cohort six) was not included in the six-month follow-up 
and only two cohorts had completed the programme by the time of the twelve-month follow-up round of 
data collection. Consequently these findings may not be representative of the whole sample. 


Finally, the findings of this evaluation also relate to a particular geographical location, primarily 
Plymouth, which limits the generalisation of the findings to a wider population. In particular, one aspect 
not addressed by this evaluation was the prevalence of other parenting programmes in the Plymouth 
area—such as the Incredible Years, parts of which were used in the development of SPOKES)—that 
may also have had an impact on outcomes. 


Future research and recommendations 


Positive effects were found on the Key Stage 1 literacy assessments for children who had reached the 
end of KS1, and on some medium term outcomes for boys. While these results did not include all cohorts 
of children involved in the evaluation, they do suggest that wider outcomes should be measured as they 
imply that the greatest added value of the programme might be in developing the wider literacy skills 
used at KS 1 rather than those measured by the primary outcomes. The positive follow-up outcomes for 
a subsample of boys in this study (at six- and twelve-month follow up), taken in conjunction with the KS1 
results, also suggest that the SPOKES programme had a greater impact on boys than on the sample 
as a whole. Consequently, future research on the SPOKES programme would be warranted with 
particular focus on both the wider outcomes of the programme and its impact on boys’ literacy. 


Given the low level of attendance of intervention parents in this evaluation, future research should also 
consider factors relating to recruitment, attendance, and implementation fidelity. Prior to any future 
evaluation, the SPOKES programme might benefit from a ‘service design’ exercise. This would be 
particularly important in considering any possible roll-out of the programme as SPOKES sessions are 
quite expensive to run and time-intensive for parents. Definitions of ‘completers’, those who have 
completed the programme, may need to be restricted to those attending a higher number of sessions. 
If attendance was higher, and the definition of ‘completers’ required a higher attendance rate, there 
might be more impact of the intervention. 


In particular, service deliverers, the developer, and referrers, need to be clear on the recruitment process 
and who is targeted. The fact that a lot of the participating children were not ‘struggling readers’ as 
originally intended for the programme may also suggest revisions are needed to the way in which low 
attaining pupils are identified in schools (such as through the Early Years Foundation Stage profile) and 
how ‘struggling reader’ is defined. A future evaluation may benefit from following up on children’s 
comprehension skills, a more advanced and critical feature of reading, in addition to the early literacy 
skills assessed as part of this evaluation. 


Perhaps the developers and providers should try to find ways to increase the participation rate as it 
might then have greater impact on parents’ support for their children’s literacy and thus improve 
children’s reading achievement, even though in this case a dose-response relationship was not found. 
In particular, parents found the programme to be successful in improving their children’s reading, and 
they reported benefitting from the programme themselves. 


Consequently, we feel it would be worth refining the programme and evaluating it again, to include: 
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e more attention to fidelity through a more thorough process evaluation to assess both what is 
happening during SPOKES sessions and implications for scalability of the final model; and 
e wider outcome measures, with perhaps more focus on the literacy achievement of boys. 
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Appendix A: Parent information sheet and consent form 


PARENT INFORMATION SHEET 


Invitation 
Dear Parent, 


Weare delighted that your school has agreed to participate in a study of children’s literacy development starting 
in January 2013. The study is being run by Plymouth Parent Partnership, the University of Oxford and the 
University of York. We very much hope you and your child would like to take part, but before you decide, it is 
important that you understand why the study is being done and what it will involve for you and your child. 


What will happen if my child and | take part? 

If you would like to take part in the study, you will be offered participation in one of two programmes: one 
involves taking part in a Supporting Parents on Kids’ Education (SPOKES) Literacy parent group, and the other 
involves literacy packs for your family. To make a fair allocation to each programme, the selection will be decided 
at random using a computer. Whichever programme you are offered, we would visit your child in school on four 
occasions, once before the programme and three times afterwards. The visits would focus on literacy activities 
and games. We have used these activities in our research before and know that children find them fun and enjoy 
doing them. For most children each visit will consist of a short one-to-one session with a researcher. For some 
children, each visit will consist of three short sessions. Before and after both programmes, we would like to ask 
some parents a few questions about things you do with your child at home, and we hope to do this on the 
telephone at your convenience. 


What are we trying to find out? 

The study will help us to work out the best ways to help parents support their children’s reading at home, and 
what are the key aspects in children’s literacy development. The SPOKES Literacy parent programme is a 10- 
week group intervention that helps parents to develop skills to support their child’s reading. If you are offered 
this programme, you will be invited to attend 10 weekly group sessions about strategies to use when reading 
with your child. If you are offered the Literacy pack programme, we will send book packs to your family in your 
child’s book bag over a 10-week period. 


What happens to the results of the study? 

A summary of findings will be given to participants’ schools at the end of the study and will be available to 
interested families. We also aim to publish our findings in scientific journals, but this may be one or two years 
from the end of the study. No individual child will be identifiable in any published report of this work. Any 
personal information will be destroyed at the end of the study. 


Do | have to take part? 

It is entirely up to you whether or not you decide to be part of this research project and attend these sessions. If 
you do decide to participate, please keep this information sheet to refer to, and please sign the enclosed consent 
forms. You are free to withdraw from the research at any time and you do not need to give a reason. Your 
withdrawal from the study will not affect the support that your child will receive from the school. 


What should | do next? 

If you are happy for your child to participate in this research, then please fill in the enclosed form and return it 
to a Plymouth Parent Partnership staff member or your child’s teacher. If you have any questions about this 
research project then please do not hesitate to contact us, our contact details are below. 
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We look forward to working with you and your child. 


Sincerely, 
Plymouth Parent Partnership, the Oxford Team and the York Team 


Plymouth Parent Partnership: University of Oxford: 

Mrs Angela Archer Professor Kathy Sylva 

Jan Cutting Healthy Living Centre Department of Education 

Scott Business Park 15 Norham Gardens, Oxford, OX2 6PY 
Beacon Park Rd, Plymouth PL2 2PQ Tel: 01865 274008 

Tel: 01752 258933 Email: Kathy.sylva@ education.ox.ac.uk 


Email: parentpartnership@plymouth.gov.uk 


Consent Form - Parents 


| have read and understood the information about this study, and have had the opportunity to ask questions. | 
understand who will have access to information provided and what will happen to the data at the end of this 
study | also understand that | can withdraw at any time by informing the research team of my decision. 

| understand that the study has been reviewed by, and received ethics clearance through, the University of 
Oxford Central University Research Ethics Committee and the University of York Education Ethics Committee. 

| agree to myself and my child participating in the project: 

Name: 

Child’s name: 

Child’s date of birth: 

Child’s gender: M/F 

Name of school: 


Date: 


Signature: 


Name (member of research team): 


Date: 


Signature: 
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Appendix B: Parent telephone interview schedule 


Date and time of call: ......... ee re (DD/MMIYY)  ...........:025 am/pm 
SEMI-STRUCTURED POST-TEST INTERVIEW 
Area 1: Factors influencing attendance (do not read out this line) 


Qu 1: Did you miss any of the 10 sessions? 
Qu 2: What were some of the things that made it difficult to attend? 


Qu 3: Can you think of ways which would make it easier, or more attractive, for parents to attend? 


Area 2: Parent satisfaction of the SPOKES programme (do not read out this line) 


Qu 4: What part of the programme did you like most and why? 

Qu 5: What part of the programme did you like least and why? 

Qu 6: if there was anything you could change about the programme what would it be? 

Area 3: SPOKES strategies/recommended activities implemented by the parent (do not read out 
this line) 

Qu 7: Which techniques or activities that you learned in the programme have you used most often with 


CHILD’s NAME (no prompting)? 


Qu 8: Can you give some examples of how/when you use the techniques or activities you learned (if 
they are used)? 


Qu Ya: What techniques or activities learned on the course did you try that you no longer use? 
Qu 9b: (if there is a response to 9a) Why do you no longer use them? 


Qu 10: Can you think of a short sentence to sum up your overall view of the SPOKES programme? 
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Appendix C: Consort diagrams by cohort 


Cohort 1 (Spring Term 2013) 


Excluded (n=1) 


Assessed for eligibility (nN=113) 
Did not meet inclusion 


Randomised (n=112) criteria (n=1) 


Declined to participate 


Y 


Allocated to control (n=56) 
Received allocated intervention 
(n=56) 


Allocated to intervention (n=56) 
Received allocated intervention 
(n=56) 


Tested (n=40) Tested (n=46) 

Not tested (n=16) Not tested (n=10) 

Excluded from analysis (n=9) Excluded from analysis (n=7) 
Analysed (n=31) Analysed (n=39) 


Tested (n=48) Tested (n=48) 

Not tested (n=8) Not tested (n=8) 
Excluded from analysis (n=20) Excluded from analysis (n=13) 
Analysed (n=35) 


Analysed (n=28) 
Tested (n=44) Z Tested (n=46) 
Not tested (n=12) Not tested (n=10) 


Excluded from analysis (n=7) Excluded from analysis (n=5) 
Analysed (n=37) Analysed (n=41) 


Tested (n=32) 
Not tested (n=24) 
Excluded from analysis (n=10) 


Tested (n=38) 
Not tested (n=18) 
Excluded from analysis (n=7) 


Analysed (n=22) Analysed (n=31) 
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Cohort 2 (Summer Term 2013) 


Assessed for eligibility (n=134) Excluded (n=0) 


Did not meet inclusion 
criteria (n=0) 


Randomised (n=134) 
Declined to participate 


(n=0) 


Allocated to intervention (n=67) 
Received allocated intervention 
(n=67) 


Allocated to control (n=67) 


Received allocated intervention 
(n=67) 


Tested (n=66) Tested (n=66) 

Not tested (n=1) Not tested (n=1) 

Excluded from analysis (n=4) Excluded from analysis (n=4) 
Analysed (n=62) Analysed (n=62) 


Tested (n=59) Tested (n=62) 

Not tested (n=8) Not tested (n=5) 

Excluded from analysis (n=3) Excluded from analysis (n=6) 
Analysed (n=56) Analysed (n=56) 


Tested (n=55) Tested (n=59) 

Not tested (n=12) Not tested (n=8) 

Excluded from analysis (n=4) Excluded from analysis (n=0) 
Analysed (n=51) Analysed (n=59) 


Tested (n=49) Tested (n=56) 

Not tested (n=18) Not tested (n=11) 

Excluded from analysis (n=3) Excluded from analysis (n=1) 
Analysed (n=46) Analysed (n=55) 
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Cohort 3 (Autumn Term 2013) 


Assessed for eligibility (n=114) Excluded (n=5) 


Did not meet inclusion 
criteria (n=0) 


Randomised (n=109) 
Declined to participate 


(n=5) 


Allocated to intervention (n=54) Allocated to control (n=55) 
Received’ allocated intervention Received allocated intervention 
(n=54) (n=55) 


Tested (n=52) Tested (n=53) 

Not tested (n=2) Not tested (n=2) 

Excluded from analysis (n=2) Excluded from analysis (n=0) 
Analysed (n=50) Analysed (n=53) 


Tested (n=47) Tested (n=51) 

Not tested (n=7) Not tested (n=4) 

Excluded from analysis (n=4) Excluded from analysis (n=1) 
Analysed (n=43) Analysed (n=50) 


Tested (n=43) Tested (n=50) 
Not tested (n=11) Not tested (n=5) 


Excluded from analysis (n=3) Excluded from analysis (n=1) 
Analysed (n=40) Analysed (n=49) 
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Cohort 4 (Spring Term 2014) 


Assessed for eligibility (n=158) 


Randomis¢d (n=157) 


Excluded (n=1) 


Did not meet inclusion 
criteria (n=0) 


Declined to participate 
(n=1) 


Allocated to intervention (n=78) 
Received allgcated intervention 
(n=78) 


Allocated to control (n=79) 
Received allocated intervention 
(n=79) 


Tested (n=75) 
Not tested (n=3) 

Excluded from analysis (n=10) 
Analysed (n=65) 


Tested (n=79) 
Not tested (n=0) 
Excluded from analysis (n=2) 
Analysed (n=77) 


Tested (n=60) 

Not tested (n=18) 

Excluded from analysis (n=2) 
Analysed (n=58) 


Tested (n=79) 

Not tested (n=0) 

Excluded from analysis (n=2) 
Analysed (n=77) 


Tested (n=57) Tested (n=68) 

Not tested (n=21) Not tested (n=11) 

Excluded from analysis (n=11) Excluded from analysis (n=2) 
Analysed (n=46) Analysed (n=66) 
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Cohort 5 (Summer Term 2014) 


Assessed for eligibility (nN=131) 


Randomised (n=128) 


Excluded (n=3) 


Did not meet inclusion 
criteria (n=0) 


Declined to participate 
(n=) 


Allocated to intervention (n=63) 
Received allocated intervention 


Allocated to control (n=65) 
Received allocated intervention 
(n=65) 


Tested (n=58 
Not tested (n=5) 

Excluded from analysis (n=7) 
Analysed (n=51) 


Tested (n=60) 
Not tested (n=5) 
Excluded from analysis (n=2) 
Analysed (n=58) 


Tested (n=60) 

Not tested (n=5) 
Excluded from analysis (n=7) 
Analysed (n=53) 


Tested (n=54) 
Not tested (n=9) 
Excluded from analysis (n=5) 
Analysed (n=49) 


Tested (n=55) 

Not tested (n=10) 

Excluded from analysis (n=3) 
Analysed (n=52) 


Tested (n=48) 
Not tested (n=15) 

Excluded from analysis (n=8) 
Analysed (n=40) 
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Cohort 6 (Autumn Term 2014) 


Received allocated 
(n=84) 


Tested (n=78) 
Not tested (n=6) 


Analysed (n=69) 


Tested (n=64) 
Not tested (n=20) 


Analysed (n=60) 
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Allocated to intervention (n=84) 
intervention 


Excluded from analysis (n=9) 


Excluded from analysis (n=4) 


Assessed for eligibility (nN=171) 


Randomised (n=168) 


Allocated to control (n=84) 
Received allocated intervention 
(n=84) 


Tested (n=74) 
Not tested (n=10) 

Excluded from analysis (n=4) 
Analysed (n=70) 


Tested (n=66) 
Not tested (n=18) 

Excluded from analysis (n=4) 
Analysed (n=62) 
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Excluded (n=3) 


Did not meet inclusion 
criteria (n=1) 


Declined to _ participate 
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Appendix D: Outcomes according to gender 


Table D.1: Primary Outcomes at post-test (all eligible boys) 


N (missing) Mean in Effect 
(95% model Size 
Cl) (Int, (95% 
Con) Cl) 


Letter ID 107.06 106.73. 7.73 
(105.73, (105.69, 
108.39) 107.77) 
Word ID 120.91 12.87 215 119.25 12.65 
(118.86, (12) (117.52, 
122.95) 120.97) 
Word 118.86 118.04 8.51 
Attack (117.47, (116.88, 
120.25) 119.20) 


Table D.2: Secondary Outcomes — social and behavioural measures at post-test (all eligible boys) 


Intervention Control 


N Mean SD N Mean SD Nin Effect 
(missing) (95% (missing) (95% model Size 
Cl) Cl) (Int, (95% 
Con) Cl) 


Prosocial 
Score 


Total 
Difficulties 
Score 


Externalising 
Score 


Internalising 
Score 


Impact Score 73 
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Supporting 
Frequency 


Supporting 
Problematic 


Proactive 
Frequency 


Proactive 
Problematic 


Total 
Frequency 


Total 
Problematic 


Table D.3: Primary Outcomes at 6-month follow-up (all eligible boys) 


Intervention Control 


N (missing) Mean SD Nin Effect 
(95% model Size 
Cl) (Int, (95% 
Con) Cl) 
Letter ID 103.72 : 102.44 7.85 
(102.49, (101.20, 
104.95) 103.68) 
Word ID 121.23 146 117.81 14.24 
(118.63, (115.45, 
123.83) 120.16) 
Word 118.86 9.78 160 116.76 9.75 
Attack (117.15, (9) (115.22, 
120.57) 118.30) 


* ps0.05 


Table D.4: Secondary Outcomes — social and behavioural measures at 6-month follow-up (all eligible 
boys) 


Intervention Control 


N Mean SD N Mean SD Nin Effect 
(missing) (95% (missing) (95% model Size 
Cl) Cl) (Int, (95% 


Con) Cl) 
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Prosocial 
Score 


Total 
Difficulties 
Score 


Externalising 
Score 


Internalising 
Score 


Impact Score 56 


Supporting 42.09 
Frequency 


43.50) 
Supporting 
Problematic 


Proactive 
Frequency 


Proactive 
Problematic 


Total 
Frequency 


Total 
Problematic 


* ps0.05 


(40.69, 


5.71 


80 41.84 
(40.79, 
42.88) 


Table D.5: Primary Outcomes at 12-month follow-up (all eligible boys) 


4.68 
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Intervention Control 


N (missing) Mean 
(95% 
Cl) 
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SD 


N Mean 


(missing) (95% 
Cl) 


SD 
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Letter ID : 94.75 8.31 
(92.50, 
96.99) 
111.26 105.83 15.26 
(106.41, (101.06, 


116.12) 110.60) 
115.55 108.26 14.77 
(112.08, (104.21, 
119.01) 112.32) 


* ps0.05 


Table D.6: Secondary Outcomes — social and behavioural measures at 12-month follow-up (all eligible 
boys) 


Intervention Control 


N Mean SD N Mean SD Nin Effect 
(missing) (95% (missing) (95% model Size 
Cl) Cl) (Int, (95% 
Con) Cl) 


Prosocial 
Score 


Total 
Difficulties 
Score 


Externalising 
Score 


Internalising 
Score 


Impact Score 


Supporting 42.62 5.96 41.09 7.05 
Frequency (40.28, (38.08, 
44.95) 44.10) 


Supporting 
Problematic (3) 


Proactive 
Frequency 
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Proactive 
Problematic 


Total 
Frequency 


Total 
Problematic 
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81.23 
(76.46, 


85.99) 


11.17 
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Table D.7: Year 1 Phonics results (all eligible boys) 


Intervention Control 


N (missing) Mean SD N Mean SD 
(95% (missing) (95% 


Cl) Cl) 


Year 1 149 29.87 9.93 170 29,21 10.19 
Phonics (13) (28.24, (8) (27.64, 
31.49) 30.77) 


Table D.8: KS1 Assessment results (all eligible boys) 


N Mean SD Nin Effect 
(missing) (missing) (95% model _ Size 
Cl) (Int, (95% 
Con) Cl) 


KS1 62 
Assessment (10) 
Reading/Writing 
Score 

KS1 62 
Assessment (10) 
Reading Score 


* p<0.05 
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Table D.9: Primary Outcomes at post-test (all eligible girls) 


N (missing) Mean SD Nin Effect 
(missing) 


(95% model Size 
Cl) (Int, (95% 
Con) Cl) 
Letter ID 108.81 : 108.06 8.56 
(107.52, (106.59, 
110.09) 109.53) 
121.78 13.93 129 121.08 13.25 
(119.54, (14) (118.74, 
124.01) 123.41) 
119.03 : 118.89 9.47 
(117.60, (117.24, 
120.45) 120.55) 


Table D.10: Secondary Outcomes — social and behavioural measures at post-test follow-up (all eligible 
girls) 


Intervention Control 


N Mean SD N Mean SD N in Effect 
(missing) (95% (missing) (95% model _ Size 
Cl) Cl) (Int, (95% 
Con) Cl) 


Prosocial 7.55 2.37 121 
Score (7.13, 
7.96) 


Total 
Difficulties 
Score 


Externalising 
Score 


Internalising 
Score 


Impact Score 47 


Supporting 81 41.06 4.89 79 41.57 6.38 144 -0.03 
Frequency (7) (39.98, (9) (40.13, (74, (-0.34, 
42.15) 43.00) 70) 0.28) 
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Supporting 
Problematic 


Proactive 
Frequency 


Proactive 
Problematic 


Total 
Frequency 


Total 
Problematic 
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Table D.11: Primary Outcomes at 6-month follow-up (all eligible girls) 


N (missing) Mean SD Nin Effect 
(missing) 


(95% model Size 
Cl) (Int, (95% 
Con) Cl) 
Letter ID 104.72 : 103.98 7.91 
(103.54, (102.47, 
105.90) 105.49) 
119.45 12.16 102 119.72 13.77 
(116.99, (13) (116.99, 
121.91) 122.44) 
119.03 : 118.55 10.05 
(117.35, (116.64, 
120.71) 120.47) 


Table D.12: Secondary Outcomes — social and behavioural measures at 6-month follow-up (all eligible 
girls) 


Intervention Control 


N Mean SD N Mean SD N in Effect 
(missing) (95% (missing) (95% model _ Size 
Cl) Cl) (Int, (95% 
Con) Cl) 


Prosocial 
Score 


Total 
Difficulties 
Score 


Externalising 
Score 


Internalising 
Score 


Impact Score 


Supporting 49 42.43 5.32 6.06 95 0.17 
Frequency (6) (40.91, (39.72, (43, (-0.21, 
43.95) 42.90) 52) 0.55) 
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Supporting 
Problematic 


Proactive 
Frequency 


Proactive 
Problematic 


Total 
Frequency 


Total 
Problematic 
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Table D.13: Primary Outcomes at 12-month follow-up (all eligible girls) 


N (missing) Mean SD Nin Effect 
(missing) 


(95% model Size 
Cl) (Int, (95% 
Con) Cl) 


Letter ID 


111.68 14.92 28 112.29 15.42 
(106.04, (6) (106.46, 
117.32) 118.11) 
113.47 12.90 36 116.06 14.11 
(109.05, (7) (111.35, 
117.89) 120.76) 


Table D.14: Secondary Outcomes — social and behavioural measures at 12-month follow-up (all eligible 
girls) 


Intervention Control 


N Mean SD N Mean SD N in Effect 
(missing) (95% (missing) (95% model _ Size 
Cl) Cl) (Int, (95% 
Con) Cl) 


Prosocial 
Score 


Total 
Difficulties 
Score 


Externalising 
Score 


Internalising 
Score 


Impact Score 


Supporting 12 43.67 5.57 11 40.91 4.89 
Frequency (4) (40.45, (1) (37.96, 
46.88) 43.86) 
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Supporting 
Problematic 


Proactive 
Frequency 


Proactive 
Problematic 


Total 
Frequency 


Total 
Problematic 
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Table D.15: Year 1 Phonics results (all eligible girls) 


Intervention Control 


N (missing) Mean SD N Mean SD 
(95% (missing) (95% 
Cl) Cl) 


Year 1 136 30.18 10.10 124 30.91 10.58 
Phonics (20) (28.45, (14) (29.01, 
31.92) 32.81) 


Table D.16: KS1 Assessment results (all eligible girls) 


N Mean SD Nin Effect 
(missing) (missing) (95% model _ Size 
Cl) (Int, (95% 
Con) Cl) 


KS1 52 
Assessment (12) 
Reading/Writing 
Score 


KS1 52 
Assessment (12) 
Reading Score 
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Appendix E: Outcomes for children with low baseline scores 
(95% of mean or less) 


Table E.1: Primary Outcomes at pre-test (all eligible pupils with 95% of mean or less at baseline) 


N (missing) Mean SD Nin Effect 
(95% model Size 
Cl) (Int, (95% 
Con) Cl) 


Letter ID 104.69 105.15 8.31 
(102.86, (103.54, 
106.51) 106.76) 
Word ID 114.83 107 115.96 11.57 
(111.80, (113.72, 
117.85) 118.20) 
Word 115.43 115.36 9.39 
Attack (113.39, (113.53, 
117.48) 117.18) 


Table E.2: Secondary Outcomes — social and behavioural measures at post-test (all eligible pupils with 
95% of mean or less at baseline) 


Intervention Control 


N Mean SD N Mean SD Nin Effect 
(missing) (95% (missing) (95% model _ Size 
Cl) Cl) (Int, (95% 
Con) Cl) 


Prosocial 6.42 2.75 96 
Score (5.76, 
7.08) 
Total 
Difficulties 
Score 


Externalising 
Score 


Internalising 
Score 


Impact Score 
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Supporting 
Frequency 


Supporting 
Problematic 


Proactive 
Frequency 


Proactive 
Problematic 


Total 
Frequency 


Total 
Problematic 


Table E.3: Primary Outcomes at 6-month follow-up (all eligible pupils with 95% of mean or less at 
baseline) 


Mean in Effect 
(95% model Size 
Cl) (Int, (95% 
Con) Cl) 
Letter ID : 101.45 8.31 
(99.69, 
103.21) 
Word ID 114.36 14.58 79 113.67 13.61 
(110.43, (2) (110.61, 
118.29) 116.73) 
Word 115.76 12.13 89 115.10 10.63 
Attack (112.71, (2) (112.85, 
118.82) 117.35) 


Table E.4: Secondary Outcomes — social and behavioural measures at 6-month follow-up (all eligible 
pupils with 95% of mean or less at baseline) 


Intervention Control 


N Mean SD N Mean SD Nin Effect 
(missing) (95% (missing) (95% model Size 
Cl) Cl) (Int, (95% 


Con) Cl) 
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Prosocial 
Score 


Total 
Difficulties 
Score 


Externalising 
Score 


Internalising 
Score 


Impact Score 


Supporting 43.74 5.20 45 
Frequency (41.74, 


45.74) 
Supporting 
Problematic 


Proactive 
Frequency 


Proactive 
Problematic 


Total 
Frequency 


Total 
Problematic 


* ps0.05 
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41.40 5.33 
(39.81, 
42.99) 


Table E.5: Primary Outcomes at 12-month follow-up (all eligible pupils with 95% of mean or less at 


baseline) 


Intervention Control 


N (missing) Mean SD N 
(95% (missing) (95% Size 


Cl) 
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Mean SD i Effect 


Cl) (95% 


Cl) 
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Letter ID 


106.93 
(100.25, 


113.60) 

110.32 105.68 
(104.88, (100.46, 
115.76) 110.89) 


Table E.6: Secondary Outcomes — social and behavioural measures at 12-month follow-up (all eligible 
pupils with 95% of mean or less at baseline) 


Intervention Control 


N Mean SD N Mean SD Nin Effect 
(missing) (95% (missing) (95% model Size 
Cl) Cl) (Int, (95% 
Con) Cl) 


Prosocial 
Score 


Total 
Difficulties 
Score 


Externalising 
Score 


Internalising 
Score 


Impact Score 


Supporting 41.50 8.60 39.92 8.57 
Frequency (34.48, (35.17, 
48.52) 44.68) 


Supporting 
Problematic 


Proactive 
Frequency 


Proactive 
Problematic 
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Total 82.00 17.08 77.62 15.55 
Frequency (68.06, (68.99, 
95.94) 86.24) 


Total 
Problematic = (Q) 
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Appendix F: Outcomes’ according to programme 
‘completion’ 


Table F.1: Primary Outcomes at post-test (all eligible ‘completers’ and control pupils) 


N (missing) Mean SD Nin Effect 
(95% model Size 
Cl) (Int, (95% 
Con) Cl) 
LetterID 193 108.37 : 107.24 8.07 
(107.24, (106.38, 
109.50) 108.09) 
Word ID 122.16 12.52 344 119.93 12.89 
(120.34, (26) (118.54, 
123.99) 121.32) 
Word 119.52 : 118.36 8.88 
Attack (118.24, (117.41, 
120.81) 119.32) 


N (missing) Mean in Effect 
(95% model Size 
Cl) (Int, (95% 
Con) Cl) 
Letter ID 104.23 : 103.07. 7.90 
(103.18, (102.11, 
105.28) 104.03) 
Word ID 121.72 13.12 248 118.59 14.05 
(119.47, (22) (116.81, 
123.96) 120.38) 
Word 119.50 . 117.49 9.90 
Attack (117.92, (116.29, 
121.07) 118.70) 


Table F.3: Primary Outcomes at 12-month follow-up (all eligible ‘completers’ and control pupils) 


Intervention Control 


N (missing) Mean SD N Mean SD Nin Effect 
(95% (missing) (95% model Size 
Cl) 


Cl) (Int, (95% 


Con) Cl) 
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Letter ID 
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111.72 


(106.91, 
116.52) 
115.37 

(111.72, 
119.01) 


108.45 
(104.71, 
112.19) 
111.42 
(108.25, 
114.58) 


15.54 


14.93 
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Table F.4: Year 1 Phonics results (all eligible ‘completers’ and control pupils) 


Intervention Control 


N (missing) Mean SD N Mean SD 
(95% (missing) (95% 
Cl) Cl) 


Year 1 164 31.03 9.55 29.93 10.37 
Phonics (15) (29.54, (28.72, 
32.52) 31.14) 


N Mean SD N Mean SD Nin Effect 
(missing) (95% (missing) (95% model _ Size 
Cl) Cl) (Int, (95% 
Con) Cl) 


KS1 64 14.55 ; 13.08 
Assessment (11) (13.69, (12.41, 
Reading/Writing 15.40) 13.75) 
Score ; 
KS1 64 15.38 : 13.58 
Assessment (11) (14.45, (12.87, 
Reading Score 16.30) 14.28) 


*ps0.05 
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Appendix G: End of Programme evaluations (Cohorts 3-6) 


We would like to know what you thought of the SPOKES Literacy Programme. The following 
questionnaire is part of our evaluation of SPOKES and your views will help us to improve the 
programme we offer. Please answer as honestly as possible. Your co-operation is greatly 
appreciated and all responses are strictly confidential. 


Name: Date: 


A: Qverall SPOKES programme 


Please indicate how enjoyable you found the overall SPOKES programme. Please tick the box 
below the smiley face that best represents how you feel. 


Ii, Overall, how enjoyable did you find the SPOKES programme? 


HOO © © 


0 (0%) 0 (0%) 1 (1%) 20(%16) | 101 (83%) 


B: SPOKES techniques/content 


Please indicate how usefulyou found the specific techniques. Please circle the response that best describes 
what you think (1= not at all useful, 5= very useful) 


1. Talking about the book before reading with your child 


1=not at all 5=very 


0(0%) | 2(2%) | 12(10%) | 32 (26%) | 78 (63%) 


2. Pausing and prompting if your child makes a mistake when reading 
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1=not at all 
useful 
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1 


2 


3 


4 


5 


5=Vvery 
useful 


0 (0%) 


0 (0%) 


3 (2%) 


14 (11%) 


107 (86%) 


Praising your child for getting a word right or for ‘having a go’ at a new word 


1=not at all 
useful 


Using Environmental Print 


1=not at all 
useful 


Encouraging writing at home 


1=not at all 
useful 


1=not at all 
useful 


1=not at all 
useful 


5=Vvery 
‘ é : 5 useful 
1(1%) 0 (0%) 1 (1%) 9 (7%) 
5=Vvery 
: : : 5 useful 
1(1%) 3 (3%) 24 (21%) | 40 (384%) | 34 (91%) 
5=Vvery 
: : : 5 5 useful 
0 (0%) 0 (0%) 3 (2%) 30 (25%) | 88 (73%) 
Using letters and sounds to read 
5=Vvery 
: si 5 useful 
0 (0%) 0 (0%) 10 (8%) | 30 (24%) | 83 (67%) 
Setting aside a specific time for reading 
5=Vvery 
: 2 : 5 useful 
0 (0%) 1 (1%) 8 (7%) 23 (19%) | 91 (74%) 


Playing with your child 
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1=not at all 5=very 
useful 1 2 3 4 5 useful 


0 (0%) 1(1%) | 5(4%) | 22(18%) | 95 (77%) 


SB Visiting the library 


1=not at all 5=very 
useful 1 2 3 4 3 useful 


2(2%) | 2(2%) | 16(15%) | 32 (30%) | 56 (52%) 


10. One-to-one visit 
1=not at all 5=very 
useful 1 2 3 4 3 useful 


2(3%) | 1(1%) | 6(8%) | 19(25%) | 47 (63%) 


Any other comments about the SPOKES techniques: 


C: Group sessions 


Please indicate how useful you found each of the following methods. Please circle the response that 
most clearly describes your opinion. (1= not at all useful, 5= very useful) 


1. Information provided by the group leader 
1=not at all 5=very 
useful 1 2 7 : 5 useful 


0(0%) | 0(0%) | 3(2%) | 12 (10%) | 108 (88%) 


2. Demonstration of skills through videos 


1=not at all 5=very 


0(0%) | 1(1%) | 18(15%) | 39 (32%) | 64 (52%) 
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Group discussion of skills 


5=Vvery 


1=not at all 3 
useful 


useful 1 2 4 5 


0(0%) | 0(0%) | 0(0%) | 22 (18%) | 100 (82%) 
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4. Role plays and practising during the group sessions 
1=not at all 5=Vvery 
useful 1 2 33 5 useful 
0 (%) 2 (2%) 11 (9%) | 30 (24%) 80 (50%) 
5. Practising the reading and other literacy skills with your child at home 
1=not at all 5=very 
useful 1 2 3 zs 5 useful 
0 (0%) 0 (0%) 1 (%) 12 (%) 110 (%) 


Any other comments about the group sessions: 
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D: Group leaders 


Please tell us your views about your group leaders. Please circle the response that best describes 


how you feel. (1 = low, 5 = high) 


1. The overall leading of the group 
1=unhelpful manent 
1 2 3 4 5 helpfiil 
0 (0%) 0 (0%) 1 (1%) 8(7%) | 113 (93%) 
2. The leaders’ preparation 
1=unhelpful S=very 
1 2 3 4 5 helpfiil 
0 (0%) 0 (0%) 1 (1%) 10 (8%) | 112 (92%) 
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3. Regarding the leaders’ interest in me and my child, I was.... 
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Anyopther comments about the group leaders: 
satisfied : : : 


4 


5 


5=Vvery 
satisfied 


0 (0%) 


0 (0%) 


1(1%) 


10 (8%) 


112 (92%) 


E: My child 


Please tell us about your child's reading. Please circle the response that best describes how you feel. (A= 
got much worse, 5= got much better) 


1. Since beginning the literacy programme my child’s reading has...... 
1=unhelpful 5=very 
, ‘ ? : 5 helpfil 
0 (0%) 0 (0%) 6 (5%) 48 (40%) | 68 (57%) 


2. Since the beginning of the programme my ability to help my child read has... 


1=got much 
worse 


1 


2 


3 


4 


5 


5=got much 
better 


0 (0%) 


0 (0%) 


2 (2%) 


28 (23%) 


93 (76%) 
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Any other comments about your child's reading: 


F: General 


Finally, we would be grateful if you would answer a few general questions about SPOKES: 


1. Which part of the programme was mast helpful to you? 


2. Which part of the programme was least helpful to you? 


3. What was most enjoyable? 


4. What did you like least about the programme? 


5. How could the programme have been improved to help you more? 


6. What would you say to a new parent thinking of participating in SPOKES? 


7. Do you have any other comments? 


Thank you for completing this form! 
The SPOKES team 
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Appendix H: Weekly Evaluation forms 


SPOKES Literacy Programme 


Your comments on today's session 


Please rate below your views on today's session (1 = low, 5 = high) 


Did you find today's session enjoyable? 
1 2 3 4 5 


Overall how useful was today's topic to you? 


1 Z 3 4 fs) 


What did you like most about today's session? 


What did you like least about today's session? 


1 would like more information in future sessions on............ 


Comments 


Name: 


Date: 
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Appendix F: Cost rating 


Cost ratings are based on the approximate cost per pupil per year of implementing the intervention over 
three years. Cost ratings are awarded using the following criteria. 


Cost Description 
£ Very low: less than £80 per pupil per year. 


EE Low: up to about £200 per pupil per year. 
EEE Moderate: up to about £700 per pupil per year. 
EEEE High: up to £1,200 per pupil per year. 
EELELE Very high: over £1,200 per pupil per year. 
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Appendix G: Padlock rating 


7 July 2016 Complete by Elena Rosa Brown 


2. Power: 5. Threats to 
1.Design: ee 3. Attrition: 4. Balance: aay Pea 
h PLAS FSS i : 
What is the ae What is the Adjustment a 


minimum Adjustment to 
quality of the (-AV/-) Me) el ge) e) to rating to 
fol} ¢-eie-]e) (= account for issues 
design of the out from the account for 
aation? effect at the are balance? with 
evaluation? valuation: ance? 
= start? horde - m interpretation? 
Evaluation design Implementation Analysis and interpretation 


Rating 1. Design 2. Power 3. Attrition 4. Balance 5. Threats to 
(Le) ==)) WEViCoTi ay 


Fair and clear experimental 

design (RCT, RDD) : < 20% 
Well-matched comparison 

(quasi-experiment) : 


Matched comparison é 

(quasi-experiment) . < 40% 

Comparison group with ; 

poor or no matching ; < 50% 

No comparator 0 Imbalanced on Poe 


The final security rating for this trial is 3 @. 
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You may re-use this document/publication (not including logos) free of charge in any format or 
medium, under the terms of the Open Government Licence v2.0. 


To view this licence, visit wwww.nationalarchives. gov.uk/doc/open-government-icence/version/2 or 


email: psi@nationalarchives.gsi.gov.uk 


Where we have identified any third party copyright information you will need to obtain permission 
from the copyright holders concerned. The views expressed in this report are the authors’ and do 
not necessarily reflect those of the Department for Education. 


This document is available for download at www .educationendowmentfoundation.org.uk 
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